I’m the associate product owner working on the project to improve the Healthy Start scheme. As part of the beta phase, it is important that we engage with users to understand their needs.

Ongoing research has helped our project team to learn about users’ shopping habits and their awareness and use of digital. We have tested prototypes of the online application form with real users to inform future iterations. 

We confirmed our target users

Previous research allowed us to identify who our target groups were:

• users who would find it difficult to use a digital service – for example due to language barriers, limited access to the internet or learning difficulties
• pregnant women eligible for Healthy Start but not currently in receipt of vouchers 
• parents already in receipt of vouchers
• the above users with English as a second language 

We made a plan for our research

To focus on the most time critical activities, we set up a 10 minute daily user research chat with our team and assigned action owners straight away. This allowed us to look at priority tasks for the day, but also highlight priorities for later on.

Our team decided to conduct user research every 2 weeks, face to face with users if possible, and to try some remote sessions in between. We used Confluence, an online tool that helps teams to share knowledge efficiently, to visualise our plan and asked everyone’s actions to be kept up to date daily.

Recruiting the right people

We reached out to local authorities and children’s centres in target regions to help us find suitable participants. We’ve received great support from a network of local authorities who have helped us to carry out research in a spread of geographical locations.

One of the family centres where we carried out research

Supporting the recruitment with resources

We supplied each venue with a screener sheet to fill in and send back to us. This meant we could find suitable participants to represent our target personas. We made sure that personal data was safely stored on a secure NHS server and was not shared outside the project team.

We also created a simple, bespoke flyer for centres to display in suitable locations or to post on social media.

These resources helped us to explain the purpose of the research, who we were looking for and how they could get involved. 

How did we conduct our research?

Surveys

Surveys are a great way of getting a high number of responses easily. Our team has conducted surveys to get views on:

• a service name
• current Healthy Start recipient views
• retailer experiences
• processes and challenges from a healthcare professional’s point of view

Our increasing network of contacts, and willingness from users to help, has meant the response rates have been great. After completing the survey, we have also received offers to be involved in helping with further research. 

Face to face chats

Results from surveys have been combined with qualitative insights from visits to children’s centres around the country. We are really grateful for the co-operation and it’s been encouraging to see that so many people are willing to help with research. 

The familiarity of visiting families in their local community has helped participants to feel relaxed and openly share thoughts on the current scheme and improvements. 

Remote prototype testing

Our remote testing has been widely spread across the UK.  We used a video link through the Silverback app. Again, children’s centre staff have helped us find those users willing to have video chats with the team. 

What challenges did we face?

As with many on site testing sessions, we did have a few ‘no shows’. Due to the nature of this research, we recognise that some of our user groups may be living in vulnerable situations or have chaotic lifestyles that meant they were unable to join us.

For our users with English as a second language, we made arrangements for an interpreter to join us to interview women from the Syrian community. We gained some real insight into family life, the cultural differences and the challenges the women faced, sometimes daily, when doing their shopping.

One of our users explained that due to anxiety they preferred not to use the telephone or video process for the chat. We adapted the session so that the user could opt out of the video/telephone call but still run through the prototype application form and provide feedback over instant chat. We also added in a few questions about their experience of digital and audio services. 

Our team carried out face to face sessions in children’s centres across the country - Plymouth, Leicester, Southend, Kennington and Leeds - we coupled this with remote testing and telephone interviews around the UK.

This map shows all project research locations and demonstrates how our recruitment strategy helped the wider team to maximise cover across the UK

So, what's next?

We are already widening our search to involve users from different parts of the UK including Northern Ireland and Wales. 

We are engaging with stakeholders including retailers from large supermarkets, convenience stores and market traders. This will give different types of service users a voice to share the impact of the scheme on them, give their ideas and highlight any potential challenges. 

To combine with the healthcare professionals survey, the team are arranging one to one interviews with health visitors and midwives to hear views of how the current scheme is promoted. This will also help us to understand how changes to the scheme may have an impact on their working day and the people in their care.

We are always keen to hear from those passionate about Healthy Start, healthy eating, obesity, vitamins or other linked subjects. To get in touch you can email us at: nhsbsa.healthyfoodbeta@nhs.net

From: NHSX

Assessment date: 6 June 2019

Stage: Alpha

Result: Met

Service provider: Public Health England

Service Description

The UK National Screening Committee advises ministers and the NHS in the 4 UK countries about all aspects of health screening and supports implementation of screening programmes.

The Committee has a list of 109 policy recommendations which are reviewed on a triennial basis. Their recommendations are based on evidence reviews which are commissioned and managed by the evidence team, and on the opinions of external stakeholders whose input is polled during public consultations. The service captures all this work by way of displaying the full list of policy recommendations and active consultations to the public.

Service users

The users of this service are:

• Evidence review managers: commission and preside over evidence reviews
• Policy stakeholders: access screening information, make statements on behalf of the perspective of their organisation
• Interested members of the public: access screening information online

1. Understand user needs

Decision

The team met point 1 of the Standard.

What the team has done well

• Usability testing in almost every sprint in alpha
• In the alpha period after the first assessment the team used an agency to recruit users with accessibility and access needs
• Tested admin interface with users at their place of work
• Tested outside of England, in Wales and Scotland
• Included staff users in almost every sprint
• Tested with members of the public who weren’t familiar with the service

What the team needs to explore

• It would be useful to understand the end to end journey for the user, what other services do they interact with to complete their wider goal? Is there anything the team learnt in their research that could be shared with these other services? Are there blockers to completing the user’s wider goal that the team could influence?
• Explore how easy to understand and accessible the content written by policy team is likely to be. Perhaps test the content with users and have policy content writers observe? Also, are the plain English summaries produced by third parties accessible and easy to understand? If this hasn’t been tested it would be interesting to explore rather than assume.

2. Do ongoing user research

Decision

The team met point 2 of the Standard.

What the team has done well

• The team have a thorough plan for ongoing user research and a dedicated user researcher in the team

What the team needs to explore

• The team mentioned they plan to do unmoderated testing (by giving the prototype to users for feedback later). This is a great way to reach more people. However to get more useful results it would help to give users structure on what they are looking for so that you don’t end up with opinions or vague reviews. You also have to weigh this evidence against data you have collected through observation. For example, you may find that evidence collected in an unmoderated test is ambiguous or confusing. If this is the case you could hold shorter follow up interviews with users.
• It appeared that the user researcher had carried out and analysed some sessions on their own. Sometimes this is unavoidable, but we would recommend at least one other person attend to take notes, that there is at least a remote viewing option and that as many as possible team members be involved with analysis

3. Have a multidisciplinary team

Decision

The team met point 3 of the Standard.

What the team has done well

• The panel were pleased that content designers had been engaged several times throughout the extended alpha
• The team explained that the content designers came from outside the team which helped ensure there was not a conflict of interest in the research and design process
• The team are creating guidelines for the helpdesk for assisted digital users and have an alternate route of engagement in the shape of a postal form

What the team needs to explore

• Based on recommendations from the last assessment where the content designer was doing some of the user research, the current user researcher chose to be less involved with content design reviews. We would recommend that the user research and content design roles are held by different people within the team and that they work closely together to set research goals, analyse research and agree on actions. The panel was pleased to hear that the team plan on bringing in a separate content designer as part of private Beta

7. Understand security and privacy issues

Decision

The team met point 7 of the Standard.

What the team has done well

• The panel were pleased to hear that the team had plans to redirect the existing URL to reduce phishing potential
• The team mentioned that they are working on a GDPR statement on how user data will be used
• The team plan on using existing PHE login infrastructure

What the team needs to explore

• The team needs to test the service with penetration testers and make sure all the content is sanitised

12. Make sure users succeed first time

Decision

The team met point 12 of the Standard.

What the team has done well

• The panel were impressed that the team have timed task completion for admin users and have reduced this from about 60 minutes to about 5 minutes

What the team needs to explore

• The team needs to test the service on mobile and tablet. Even though the current site isn’t accessed regularly on mobile, a lot of their users are repeat visitors and may have learned to avoid using their mobile. New users, the public in particular, will expect to be able to access content and complete tasks on their preferred device
• Make changes based on the usability testing, for example the screen magnifying issue with the checkboxes. If this is a design system pattern the team could suggest a change to the design system by contacting the team

13. Make the user experience consistent with GOV.UK

Decision

The team met point 13 of the Standard.

What the team has done well

• The team plan to run workshops with policy team who write the content so that they write in the GOV.UK style
• They have engaged with other government teams who are delivering services for internal users. Keep doing this!

What the team needs to explore

• Make a plan for ongoing engagement with policy team content writers to make sure they are still writing in the GOV.UK style
• If there are health terms not covered by the GOV.UK style guide use the NHS content style guide to supplement it
• PHE staff should be able to get access to cross-gov Slack and the Google groups. Contractors without a government or NHS email address could ask questions through PHE staff in their team. Or PHE should consider giving PHE email addresses to contract staff so that they can engage with the cross-gov community
• It would improve the service to have a dedicated content designer in the team for beta, this doesn’t need to be full time if that isn’t necessary. This role could also lead on introducing content design methods to policy teams

14. Encourage everyone to use the digital service

Decision

The team met point 14 of the Standard.

What the team has done well

• Users have the option of using email, letter or telephone to complete their task
• Helpline is on every page and helpdesk can guide user the right content
• The team had done analysis on number of email sign ups that came from outside of England and matched this to the number of registered charities outside of England
• Product manager regularly reviews helpdesk reports

What the team needs to explore

• It appeared that the team have engaged with the helpdesk, but not carried out any training with them. We’d recommend training them in the new website and have regular engagement going forward

16. Identify performance indicators

Decision

The team met point 16 of the Standard.

What the team has done well

• The team have gathered some benchmark KPIs for comparison

What the team needs to explore

• Add event tags to key behaviours and actions on the site so that you can interrogate this data in Google Analytics
• Look at gathering data on task completion or click rate on old site and compare it to the same data on the new site. This would be useful when comparing changes in behaviour of mobile users or users from different traffic sources
• It might be useful to gather qualitative data from internal users on the old site and compare it to the new site once it is in beta. This could be in the form of observing people use the site, or by asking for their feedback on specific tasks

17. Report performance data on the performance platform

Decision

The team met point 17 of the Standard.

What the team has done well

• The service is registered with the performance platform team and product manager has been speaking with them

What the team needs to explore

• Following a review of other PHE services on the platform we recommend that the team look to add the service to the site

What are Healthy Start vitamins?

The Healthy Start scheme provides vouchers towards fruit, veg and milk and free vitamins for families on income based support.

The vitamins are manufactured by the Department of Health and Social Care in the form of tablets for women to take during pregnancy and drops for young children. This supports healthy bone growth and development.

Once families have applied for the Healthy Start scheme, they receive paper vouchers for both food and vitamins. However, uptake of the vitamins is low and distribution is very complex.

How does it work?

Each local authority is responsible for making Healthy Start vitamins available in designated locations. Some choose to hand them out through children’s centres, others through health professionals or pharmacies.

Although beneficiaries must present their vouchers, in some areas they are available to all pregnant women. The ordering, distribution, collection and reimbursement varies by region and involves lots of different users.

Our simplified service map of the users involved and their tasks

Why did we research vitamins?

The Healthy Start scheme is being improved. The team are making it easier for people to apply and are developing a more convenient way to purchase healthy food items. This has left a gap for what should happen with the vitamins part of the scheme.

We started with a crucial question: ‘with a prepaid card solution currently being trialled to replace paper vouchers, how will beneficiaries access their free vitamins?’

We led a vitamins discovery to understand:

• who are the users and what are their needs?
• how does it work, where are the pain points and opportunities?
• how can we take vitamins forward in line with the wider Healthy Start scheme?

Who did the research?

There were 3 of us, a product manager, delivery manager and user researcher. We worked closely with a policy expert to understand Healthy Start vitamins in just over 6 weeks.

Who are the users?

Users are the people who use or are affected by the scheme. This includes:

• beneficiaries (pregnant women or families with a child under 4 and receiving income-based support)
• local authorities
• healthcare professionals
• distribution outlets (children’s centres, pharmacies, health centres)

To learn about our users, we immersed ourselves in their environments by visiting places where they typically go. Contextual research was combined with in-depth interviews and surveys. In the discovery, our team engaged with 101 local authorities, 237 distribution outlets, 98 healthcare professionals and 931 beneficiaries in rural and urban areas.

This was a challenge with such a small team, so thank you to everyone who took part.

One of the children’s centres we visited in Kidderminster

What we discovered...

Beneficiaries struggle to access Healthy Start vitamins

Many participants were unable to collect their free vitamins. This is because the distribution outlets are too far away, it isn’t clear where to go or the vitamins are out of stock. The website lookup is out of date and people are unsure where to signpost beneficiaries.

The voucher causes confusion

There is a VOID stamp printed on the voucher when it is no longer valid. This leads to confusion and mistrust. The messaging around pricing was perceived to be misleading, especially to those with dyslexia and low literacy skills.

There's a lack of guidance for local authorities

Often one person is responsible for setting up the vitamins in their local area. Participants we spoke to had inconsistent information about licensing, few resources and were unsure where to go for help.

Health visitors are faced with other priorities

Health visitors are an important touch point during maternity. Those we spoke to felt vitamins were important, but have a high caseload and other priorities for vulnerable families such as housing or domestic abuse. This affects their ability to recommend vitamins effectively.

The process is admin heavy for distribution outlets and a short shelf life makes it difficult to predict stock and demand

Some beneficiaries don’t feel they need vitamins or aren’t aware they can get them. However, before we look at attitudes and awareness, it’s important that those already on board are able to easily access their free vitamins.

Recommendations

We proposed things to do and things to test, based on their level of impact and effort. Some of these include:

Things to do

• refresh the website content to meet the needs of each user group
• develop guidance for local authorities and clarify licensing
• update the messaging sent out to beneficiaries, explaining what vitamins are available, why they are important and how to collect them

Things to test

• online distribution to remove location barriers
• using the new prepaid card for eligibility to claim free vitamins
• making vitamins available in supermarkets as a regular touch point

Recently, a useful session took place with the Healthy Start policy team to collaboratively vote on what happens next. This focused on user needs while taking into account constraints of the Department.

We are now confident that we understand how Healthy Start vitamins works, what the users need and the current problems. This will help to inform the wider Healthy Start changes.

You can follow our progress on improving Healthy Start.

If you have any questions about the vitamins work, please contact our user researcher sophie.rankin@dhsc.gov.uk

Photo credit Zac Durant @zacdurant

Public Health England (PHE) is co-designing Our Family Health to support personalised behaviour change for families with children aged 4 to 7 years. The service aims to help whole families adopt lifestyle behaviour changes to help their children either maintain or grow into a healthy weight.

The opportunity

Families who engage with local face-to-face family weight management services benefit from that support. The trouble is that not all families with children identified as being above a healthy weight have access to such services. Even when services are available, families do not always use them or sustain change once support has ended.

PHE has a responsibility to support local services. So, it is important to understand how healthy lifestyle behaviour change support can be more accessible to families and to better meet their needs. With technology featuring in our everyday lives, it makes sense to explore how it can support families as they embark on healthy lifestyle changes.

We found that there are many digital products in the health and wellbeing market, but there is a gap and an opportunity for trusted, family friendly digital support that adequately meets families’ needs. We also found that families need solutions that bring together high-quality content and functionality in a simple way, to meet their complex needs and diverse lifestyles.

This is particularly true of families who have the motivation to get on their healthy weight journey, but don’t have the tools or support to do so. These families end up trying to make changes on their own, often with little success which can be disheartening.

How we worked

The team at PHE collate user research

Using an agile way of working, the team has combined know-how around healthy eating and healthy weight, behavioural science, and human-centred design. We spent time with parents and children nearly every week in the last year to better understand their lifestyles, motivations and challenges, to shape and prototype a service with them that really works for them.

We found that for many families, it’s about health, not weight. They often know what healthy means, but they need help with making lifestyle changes. They’re also connected through digital 24/7 and really value using technology to help with children’s learning and development.

PHE and FutureGov completed the discovery and the alpha phases between January 2018 and February 2019. Read more about how we worked in discovery  and our alpha service assessment.

The service

Our Family Health, our early stage prototype, is a healthier lifestyle service that supports whole families in the areas of healthy eating, being active, parenting and happiness.

During alpha we iteratively prototyped a range of solutions with 20 families from C2, D and E income groups  – a test and learn approach. We have learned from families that this service needs to provide the following value propositions to best meet their needs:

A range of healthy tools

The service provides practical, easily accessible family health information, activities and games ‘kite marked’ by the NHS/PHE as trusted sources, and available at hand 24/7.

“I feel like this is the kind of thing that would keep supporting you  - rather than getting dusty on the shelf like all the other things I’ve tried...this service wouldn’t abandon me!” – Parent

One simple step at a time

Families can set and monitor personal health goals. The goals are developed by experts, and broken down into achievable, practical steps the whole family can do together.

“It would be good to get those nudges via text to remind me to do something (…), rather than me feeling it’s all too hard and just giving up on it”- Parent

Seeing progress

Families can see their progress and achievements, and track behaviours over time.

“Would be good to see how we’re doing on the dashboard - if we’ve still got a bit to do before we reach our goal for today, it would be the motivation to say to the kids - ‘come on, let’s go for a 10 minute walk (…). It would get us out the door’ ”  - Parent

Others like us

Parents can join a moderated online community of parents on a similar journey for encouragement and inspiration. They can ask questions and receive instant expert-based answers through a virtual assistant.

“If you just wanted a black and white answer at 4am when you were on your own and maybe worrying about something, it would be useful (instant Q&A)” - Parent

Learning through play and reward

The service is as much for children as it is for adults. Children can access a child-safe section with fun videos and games, where their goals are linked to a second-life avatar to transfer online into real-life behaviours.

“You can change him into a different cat, you can feed him. Alien – I liked feeding,(…), washing the alien.” - Child

What's next?

The alpha phase built on findings from discovery and prioritised and tested concepts with families. The beta phase will focus on building the minimum viable product and testing that over an extended period of time with families across the country.

This future phase of the work represents an exciting opportunity – not least of all learning from families about their experiences using the service – what worked for them, what didn’t, and how we can make it better. We are now busy thinking about the skills we need at PHE to continue to improve and manage the service.

We’re also looking at how we evaluate how well the service is working for families, and how it can work alone, as well as alongside existing face-to-face services.

Email jamie.blackshaw@phe.gov.uk to learn more about this work or contribute to our understanding of the opportunity space. Look out for more blogs about this service from the team in the future.

Photo credit: Rodion Kutsaev @frostroomhead

This blog is about the work we are doing to improve the way we monitor and approve spend for services that the public and the NHS use, such as the NHS.uk website.

Getting approval for spending money

Part of the digital team's work here at the department is to look across the department and its various arms-length bodies (ALBs) to keep track of where money is being spent on digital activities and manage approval of that spend. We do this to make sure:

• services are based on user needs
• we aren’t trying to fix the same problem twice
• teams have the right resources in place to deliver services

When someone wants to develop a digital service, such as a website or a new way of delivering a benefit, they send in a form to our team describing what they plan to do. We then review and approve their plan based on a set of standards and best practice (including the service standard and technology code of practice).

There are a few downsides to this process:

• it’s an extra form to fill in
• it can delay getting a project approved
• sometimes the team don’t know they need approval

Making the process simpler

To do things better, we’ve been introducing a new way to make sure that the department’s money is being spent in the right place and in the right way: pipeline. With pipeline, instead of checking spend externally, we embed the checks in existing steps that teams would have to take anyway.

Some teams are already trialling pipeline and blogging about their experiences. We’re gradually adding more services as we go.

Pipeline is made up of 3 governance mechanisms:

1. self assessments where teams review their work against a list of 7 criteria and give themselves a rating
2. assurance boards where the department, Government Digital Service (GDS) and members from the organisation's governance come together to review and confirm self-assessment ratings
3. Joint Assurance Reviews (JARs) where senior stakeholders from the department and GDS review the pipeline and submit any projects marked as ‘control’ for approval by the minister.

What we've done so far

We have:

• made a pipeline in Google Docs and a summary view for ALBs
• added projects from 3 ALBs to date
• created a core departmental pipeline for services still going through legacy spend controls processes.

For the first time, we have the potential for a complete view of planned and current digital and technology projects across the department and our ALB community. The opportunities this provides for the department and its ALBs are very exciting.

Getting on to pipeline

If you’re part of an organisation that’s transitioning onto pipeline and want to prepare, here are some things that will make it easier:

• identify a regular review process that could be used as a platform for a self assessment. This could be a check-in or show and tell that was then built on to include pipeline questions
• think about which existing governance processes could be amended to include assurance reviews
• create and maintain a tracker that records a list of ongoing and future projects or services within the organisation.

Future ideas

While the main focus at the moment is on getting the pipeline to work for the department and its ALBs, we are also considering how to get even more value from the process. For example, as mentioned earlier in this blog, we are working on providing a view of pipeline activities across ALBs to help inform the community's work.

In addition to that we are also looking at creating a service register of activities that could be published openly to help increase transparency in the way we operate.

If you’d like to learn more about pipeline or if you have any questions please get in touch with Sam Martin at samuel.martin@nhsx.nhs.uk.

Working on the evaluation toolkit

Imagine you’re a product manager who works in public health. You lead a team in developing a digital health product for quitting smoking. The product has thousands of active users across the UK.

Your organisation wants to know what effect the product has had on people’s health, but there’s no budget available to evaluate it. The team did not build indicators in, so you’re unsure if it’s successful in achieving its intended health outcomes – primarily, quitting smoking.

Evaluation Toolkit

At Public Health England (PHE), we’re working on a project to enable PHE and the wider health system to better demonstrate the impact, cost-effectiveness and benefit of digital health products to public health.

We’re developing an evaluation toolkit, which supports product managers and the rest of their delivery team in building an evaluation strategy into their project from the start. The toolkit helps teams understand if their digital health product has achieved its intended health outcomes.

Proof of Concept

During the alpha phase of this project, we tested the value proposition behind the evaluation toolkit by supporting the PHE Couch to 5K team in building their own evaluation strategy for their Couch to 5K app. We used our discovery research to define how the evaluation service should work, and which steps are crucial to the evaluation process.

Based on our findings, we defined the stages of carrying out an evaluation as:

• define health outcomes
• identify success indicators
• choose evaluation methods
• analyse data

We tested this process with the Couch to 5K team over a series of workshops, with positive results. The team found that:

• the tools and templates were useful
• the evaluation process fitted in with their workflow
• carrying out the evaluation activities as a team worked well
• having evaluation experts present was beneficial

Logic Model

Creating a logic model

A crucial part of defining your digital health product’s outcomes is creating a logic model. We tested the logic model template in the evaluation toolkit with:

• the Couch to 5K team at PHE, who used it to kick-start their evaluation journey
• the Health Checks team at PHE, who are working on prevention cardiovascular disease for 40 to 74 year olds, used it to understand their outcomes as a team
• the Vitamins project at Department of Health and Social Care (DHSC), who are carrying out a project on distributing vitamins to low income families, the logic model helped them to understand their intended health outcomes
• Digital Health Intelligence team at PHE who used it to align their team around their outcomes
• NHS Digital who used a logic model to create project specific indicators that measure the benefit of the NHS.UK platform in improving health literacy

These tests allow us to create a template that helps teams decide on the intended health outcomes of their digital health product, and how they’ll achieve them. As well as helping people define their outcomes, it also allowed teams and their wider stakeholders to align on their goals for their project. A Business Analyst in the Department of Health and Social Care shared that:

A logic model is a low risk high reward thing to do. It’s planning one session. It’s very little prep you print a bunch of words and get a bunch of people in a room. If you really don’t need it you’ll be done in 30 mins!

Usability Testing

We carried out three rounds of usability testing with people from digital delivery teams at PHE, DHSC, charities and health start-ups. Based on our findings, we decided to focus on product managers as our primary users because they:

• oversee the development of digital health products
• need to understand if the products are successful
• are responsible for facilitating evaluation within the delivery team

We learned that people were most trusting of evaluation advice which came from colleagues. As a result of this we set up online evaluation communities on Slack and KHub, to give people a space to share evaluation advice.

We also worked closely with partners at NICE, NHS Service Manual and apps library to ensure that the evaluation toolkit fits with their work and can be linked to from their platforms. This way, the evaluation advice will spread through the health system through colleagues.

In our first iterations of the prototype, we included a bank of common indicators. People could browse indicators in their subject area, to get a measure of how well they were meeting their intended health outcomes.

During our research we learned that people felt confident choosing indicators without the bank, and that indicators people chose were often very specific to their product. In the next rounds of testing we redesigned the indicators section so that it included some guidance and no bank of common indicators.

Accessibility testing

At this early stage in the design process, it was important to know whether the service would work for people with access needs. Static versions of the landing page and logic model wireframes from the evaluation toolkit were tested with users who had:

• Asperger syndrome
• partial eyesight
• hearing impairments
• learning disabilities

We learned a lot from these sessions, and made changes to the prototype:

• short, succinct content with clear, descriptive headings
• simpler language
• written descriptions to support diagrams
• the ability to print out templates

Academic support

We also carried out a number of testing sessions with academics from Edinburgh University, King’s College London and Imperial College London to get feedback on the evaluation process and understand if our explanation of evaluation is correct. The feedback we received was that we needed to define ‘evaluation’ and ‘evaluation methods’ more clearly, so we spent time tweaking these definitions until we reached an agreement. We also validated the process for evaluation, and added the ‘analyse your data’ page to the homepage, after hearing this was a key part of the evaluation process we had been missing.

Content sense checking

Throughout alpha, we worked on making the language around evaluation understandable to non-evaluation experts. We carried out sense checking sessions, where evaluation and non-evaluation experts gave feedback on the evaluation toolkit. Our findings from these exercises helped us to:

• agree on definitions of our most important terms, before we started writing and building things for users
• produce better prototypes earlier, and immediately respond to content-focused insights from early testing

Culture of evaluation

PHE hosted an evaluation event

Alongside the evaluation toolkit, we’re developing an evaluation culture at PHE. A culture that allows time for evaluation and fosters the skills needed to carry out evaluation is crucial in ensuring that evaluation is adopted.

Throughout the alpha phase, we researched and prototyped ways to build the evaluation culture at PHE. We created online channels for an evaluation community, which will continue to grow throughout the project. The Slack community immediately gained interest from people in the public health sector, with very little promotion.

We hosted an evaluation event that brought evaluators and those interested in evaluation together to share best practice. During the event, we encouraged people to share what they wanted out of an evaluation community. As the project continues, we’ll continue exploring what evaluation training could look like, building on the work done during the proof of concept.

We have held face-to-face testing sessions with delivery teams and have received positive feedback about these sessions. During usability sessions, people expressed a need to grow their evaluation skills.

For this reason, we’re further exploring the idea of evaluation training, where people can take part in a day long course on evaluation. The evaluation toolkit would support the training, which teams could continue to use after the training.

Evaluation may also form a part of DHSC’s spend controls, pipeline guidance and assurance process. We’re working to build it in to our approvals and spend control process, so that funding is distributed on the basis of health outcomes. This should incentivise teams to carry out evaluation.

Next steps

We’ll now move into private beta phase. We’ll continue working with a multidisciplinary team, bringing in academic experts in evaluation and developers to build the toolkit. Our team will continue to create an evaluation service that works for delivery teams, so they can understand the impact that their digital health product is having on users’ health outcomes.

Find out more about evaluating digital public health.

The report from the private beta assessment for NICE’s comment collection service.

Assessment by: Department of Health and Social Care

Assessment date: 3 April 2019

Stage: Private beta

Result: Met

Service provider: NICE

Service description

‘Comment collection’ provides a way for NICE stakeholders (people who the guidance effects) to comment directly on the documents produced when developing guidance. It also provides a way for teams within NICE to request these comments, set up the consultations, and collate the comments for response in a way that avoids repetitive manual handling.

Service users

Primary user groups are NICE guidance producers, coordinators and external commenters (organisations and the public).

1. Understand user needs

Decision

The team met point 1 of the Standard.

What the team has done well

The panel was impressed that:

• The team understand the needs of internal and external users of the service.
• They are capturing insights from users in a variety of ways, including usability testing, survey responses, hotjar recordings and calls from the enquiry handling team.

What the team needs to explore

Before their next assessment, the team needs to:

• Carry out usability testing more regularly and split this into rounds. For example, test with a group of participants, analyse and iterate (Round 1). Then test the iterated journey with a group of different participants (Round 2).
• Conduct observations and test usability with internal users who receive and respond to comments (NICE staff) and those who modify the guidance based on comments (Advisory Committee Members).
• There needs to be a robust way of capturing and responding to internal feedback, making sure issues are logged, analysed and prioritised in the UX and design work.
• Make sure that public and external users are not deprioritised against internal users. Particularly those who may find it hard to influence NICE guidance because of lack of easy access or digital ability.
• Consider paying incentives and / or using a recruitment agency to recruit user groups that meet your criteria and aren’t already familiar with the concept of NICE guidance. This has the added benefit of saving the user researcher’s time spent on recruitment, will encourage participation and make sure they get the right people.

2. Do ongoing user research

Decision

The team met point 2 of the Standard.

What the team has done well

The panel was impressed that:

• The user researcher acknowledged the need to do more focused usability testing going forward and they are looking at better approaches to analysis.
• The whole team have been involved in user research, observing and taking notes.
• The team plan to test more things they are aware need fixing, such as navigating from the overview page to the main content.
• There is another accessibility audit coming up.

What the team needs to explore

Before their next assessment, the team needs to:

• Conduct more accessibility testing with representative users who have hearing, motor and cognitive impairments such as dyslexia.
• Work out how to capture survey feedback from external users earlier on in the journey. The survey has a small number of responses and is only available once the user has completed a comment. This could mean it is only capturing responses from those who have been successful.
• In usability testing, provide each user group with relevant tasks/scenarios. For example, for a first-time user it would be good to explore how they arrive at the correct place, as opposed to starting all users from the overview page. What would they type in and where would this take them to? Different scenarios will help test this.
• Do more contextual user research with internal users and observe them using the service.

3. Have a multidisciplinary team

Decision

The team met point 3 of the Standard.

What the team has done well

The panel was impressed that:

• There is one service manager who leads multiple teams working on different parts of a service and/or code base.
• Team were able to respond to ad-hoc requests with mock-ups and quick testing or by working closely with users, demonstrating to senior people they were happy with the service.
• The team are co-located and pair up across disciplines, for example developers with user researchers. It’s good that developers observe user research sessions.
• There appeared to be a representative mix of genders in the team.

What the team needs to explore

Before their next assessment, the team needs to:

• Review whether it’s a good idea for content to be covered by a user researcher. Are there other content designers in NICE who could support?

4. Use agile methods

Decision

The team met point 4 of the Standard.

What the team has done well

The panel was impressed that:

• The team work well together pairing up to approach problems across User Research, UX, Design, Testing and Development.
• User research revealed findings that the team didn’t expect, which challenged their initial assumptions. One example was the use of icons, they thought these would help users but during usability testing participants didn’t understand what they meant. The team have responded by replacing icons in future iterations.
• The team have regular show and tells, sharing their work with the rest of the team.

What the team needs to explore

Before their next assessment, the team needs to:

• Consider ways of capturing issues from different sources in a centralised place. This will provide a holistic view of the service and help to prioritise what should be worked on when. For example, the user researcher has kept a thorough log from usability testing. When issues arise from survey responses and the inquiry handling calls etc. the team could combine these into one place to see the impact and frequency of issues across multiple sources.

5. Iterate and improve frequently

Decision

The team met point 5 of the Standard.

What the team has done well

The panel was impressed that:

• The service manager was able to explain the lifecycle from user research through to design and development.
• The team are internal staff and work closely together in the same space.
• The prototype has undergone various iterations throughout beta, responding to needs from user research.

What the team needs to explore

Before their next assessment, the team needs to:

• Make sure user research is happening in some capacity during every sprint.
• Once something has been tested, analyse, iterate and test again. Try to avoid iterating during testing, this is what the next round can be used for.
• Do mobile testing on different devices to understand the look and feel, page layout and usability. From the script, it looks like mobile was tested via a computer screen. This means the usability on a mobile device (for example, selecting text and leaving comments) may not have been accurately represented. If the participant has an iPhone or iPad, they can record their own screen or using Lookback, Reflector or Quicktime player the screen can be projected to a computer whilst they interact with the mobile device itself.

6. Evaluate tools and systems

Decision

The team met point 6 of the Standard.

What the team has done well

The panel was impressed that:

• There was a detailed justification for each technology choice.
• Monitoring was in place.
• Sensible and mature technology has been chosen.

What the team needs to explore

Before their next assessment, the team needs to:

• Ensure they have explored how they plan to upgrade components when new releases are available.

7. Understand security and privacy issues

Decision

The team met point 7 of the Standard.

What the team has done well

The panel was impressed that:

• There was a strong emphasis on privacy and user security.

What the team needs to explore

Before their next assessment, the team needs to:

• Consider whether personal information needs to be in the (internal) XLSX download.
• Look at whether automated tools could be used to reduce spam and remove sensitive personal details.

8. Make all new source code open

Decision

The team met point 8 of the Standard.

What the team has done well

The panel was impressed that:

• All the code was available on GitHub.
• A suitable license had been chosen.
• Security of API keys was well managed

What the team needs to explore

Before their next assessment, the team needs to:

• Understand how they can open up other internal components that they used.

9. Use open standards and common platforms

Decision

The team met point 9 of the Standard.

What the team has done well

The panel was impressed that:

• Site is in HTML5 and works across all modern browsers

What the team needs to explore

Before their next assessment, the team needs to:

• Consider the reliance on closed formats like DOCX and XLSX. Even though they are for internal use only, it still restricts the technology choices available to you.
• Ensure that all PDFs comply with PDF/A - or republish them as HTML5 as set out in GDS guidance.

10. Test the end-to-end service

Decision

The team met point 10 of the Standard.

What the team has done well

The panel was impressed that:

• There are a good range of pre-prod environments.
• The service was well tested across multiple devices - including automated testing.

11. Make a plan for being offline

Decision

The team met point 11 of the Standard.

What the team has done well

The panel was impressed that:

• The team had a sensible backup strategy.
• There are good business processes for dealing with unexpected extensions.
• There is an external facing helpdesk able to directly raise issues in JIRA.

What the team needs to explore

Before their next assessment, the team needs to:

• Test their backups - and perform a recovery exercise.
• Consider whether teams other than ops need to get alerts.

12. Make sure users succeed first time

Decision

The team met point 12 of the Standard.

What the team has done well

The panel was impressed that:

• In usability testing, the team found that there was little difference between the first time users and experienced users in being able to complete tasks.
• There is assisted digital support in place via the enquiry handling team who are trained and experienced.
• The team recognise the need for more complex features to be tested, such as the need for larger tables and different types of consultation.

What the team needs to explore

Before their next assessment, the team needs to:

• Ensure the HTML editor is accessibility tested.
• Test on different screen sizes and devices themselves (see point 5).
• Test the guidance teams uploading different types of document.
• Test all edge cases in the journey, so when the user submits an error, uploads something of the wrong format, tries to login with the wrong account etc. to make sure the experience works in all scenarios.

13. Make the user experience consistent with GOV.UK

Decision

The team met point 13 of the Standard.

What the team has done well

The panel was impressed that:

• NICE has a design system that is based on existing GDS components.
• There is a team who own the design system, but the service teams are responsible for designing and testing new components.

14. Encourage everyone to use the digital service

Decision

The team met point 14 of the Standard.

What the team has done well

The panel was impressed that:

• The team are working with internal users in private beta to make sure the service works for their needs and they can use it.
• Sponsors from internal users’ teams come to showcases and report back.

What the team needs to explore

Before their next assessment, the team needs to:

• Explore how external users (the public) with low digital access and ability can use the service.

15. Collect performance data

Decision

The team met point 15 of the Standard.

What the team has done well

The panel was impressed that:

• The team has goals set up for full user journey and can track drop off at different stages.

What the team needs to explore

Before their next assessment, the team needs to:

• Track user journeys across all parts of NICE domain (team have this on their roadmap), so that an exit to another NICE site is not treated as an ‘exit’.
• Track how people use the service over multiple sessions.
• Make sure there is someone in the team who can access and report on analytics regularly. This can be the responsibility of someone already in the team.

16. Identify performance indicators

Decision

The team met point 16 of the Standard.

What the team has done well

The panel was impressed that:

• Team have used creative methods, such as a time-motion study, to benchmark current performance.
• There is a sponsorship group who have agreed KPIs.

What the team needs to explore

Before their next assessment, the team needs to:

• Look at whether a cost/benefit analysis on potential new features will help them prioritise the roadmap.

17. Report performance data on the Performance Platform

Decision

The team met point 17 of the Standard.

18. Test with the minister

Decision

The team met point 18 of the Standard.

What the team has done well

The panel was impressed that:

• The team will be demoing the service to their board on multiple occasions.

Conditions before the next assessment

• Do further rounds of usability testing with external users and iterate the service.
• Test service with internal and external users who have access needs.
• Test on different devices and screen sizes.

The report from the alpha assessment for NHS Digital's GP IT Futures Buying Catalogue service.

Assessment by: Department of Health and Social Care and Public Health England

Assessment date: 20 March 2019

Stage: Alpha

Result: Met

Service provider: NHS Digital

Service description

The GP IT Futures Buying Catalogue is a digital marketplace for healthcare staff to purchase assured software and services from suppliers.

Service users

Primary users groups are suppliers who want to sell their services to healthcare staff and buyers of software/IT for general practice and primary care establishments.

1. Understand user needs

Decision

The team met point 1 of the Standard.

What the team has done well

The panel was impressed that:

• The team has a good understanding of their broad users groups (suppliers, buyers, and end-users) and sub-categories of core users (technical group, procurement leads, GPs’ admins, IT and practitioners). They have engaged throughout discovery and alpha with users and iterated based on their needs.
• They have shown several examples where they changed direction based on user feedback and needs. This includes learning that video demonstrations of each product/service may not work as intended.
• The team overcame some limitations/restrictions to test with some of the core users group due to commercial implications. They have compensated this by testing with proxy users and group engagements, such as webinars.

What the team needs to explore

Before their next assessment, the team needs to:

• Test end-to-end with all user groups.
• Make sure they test with those who will be managing the service in the future.

2. Do ongoing user research

Decision

The team met point 2 of the Standard.

What the team has done well

The panel was impressed that:

• The team had dedicated user research resources during discovery and most of alpha. When the user researcher left the team at the end of alpha, they compensated by sharing research duties amongst themselves. This showed great commitment to user research but should be considered a temporary solution that needs to be amended in beta.

What the team needs to explore

Before their next assessment, the team needs to:

• Fill the user researcher vacancy. The team needs to have a dedicated user researcher in place for private beta.
• The team has outlined a high-level plan for testing core capabilities in private beta for a buyers journey. This requires more thought and detail.
• Put a solid plan in place to test the current suppliers’ journey.
• Do more accessibility testing. Further research with users with diverse accessibility needs is required.

3. Have a multidisciplinary team

Decision

The team met point 3 of the Standard.

What the team has done well

The panel was impressed that:

• They had a multidisciplinary team for discovery and alpha.
• The team was full-time on the buying catalogue
• There was a 50/50 gender mix on the team

What the team needs to explore

Before their next assessment, the team needs to:

• Hire user researchers
• Bring more content design expertise into the team
• Have access to a cyber-security specialist during private beta

4. Use agile methods

Decision

The team met point 4 of the Standard.

What the team has done well

The panel was impressed that:

• They work in an agile manner.
• They use the tools one would expect of an agile team.
• We heard of cross-organisation collaboration, especially with the Digital Marketplace team.

5. Iterate and improve frequently

Decision

The team met point 5 of the Standard.

What the team has done well

The panel was impressed that:

• They have the ability to iterate and improve the service.
• They feel empowered to test ideas and cease working on things if the don’t get evidence that they are valuable.

6. Evaluate tools and systems

Decision

The team met point 6 of the Standard.

What the team has done well

The panel was impressed that:

• The team have a good mix of systems that are easily portable.
• There is excellent design for scalability.
• Cloud based tools were used.
• The team did daily updates throughout pre-production and staging environments to live.
• They have already researched alternative CRMs.

What the team needs to explore

Before their next assessment, the team needs to:

• Plan for many simultaneous users.
• Speed up page progress. Ideally to less than 3 seconds.

7. Understand security and privacy issues

Decision

The team met point 7 of the Standard.

What the team has done well

The panel was impressed that:

• The team had knowledge of different fraud vectors
• They seemed confident that their service will be secure and are following the NHS Digital’s cloud security good practice guide

What the team needs to explore

Before their next assessment, the team needs to:

• Regularly test backups.
• Explore whether two-factor authentication (2FA) is needed for users (front-end and back-end).

8. Make all new source code open

Decision

The team met point 8 of the Standard.

What the team has done well

The panel was impressed that:

• They have a Github repository and want to code in the open
• Private assets are separated.

What the team needs to explore

Before their next assessment, the team needs to:

• Continue to code in the open under an appropriate license.

9. Use open standards and common platforms

Decision

The team met point 9 of the Standard.

What the team has done well

The panel was impressed that:

• The team is using HTML5, CSS, JS
• They are using openID Connect for login

What the team needs to explore

Before their next assessment, the team needs to:

• Resolve minor usability issues (checked using HTML checker).
• Not solely use MS Excel files. Offer ODS, CSV and other open document formats.

10. Test the end-to-end service

Decision

The team met point 10 of the Standard.

What the team has done well

The panel was impressed that:

• The current iteration works with all major browsers.
• The team has evidence that users will be using desktops but "reader mode" on mobile worked well nonetheless, indicating good semantic markup.

What the team needs to explore

Before their next assessment, the team needs to:

• Ensure that the service will be upgraded with each major browser release.

11. Make a plan for being offline

Decision

The team met point 11 of the Standard.

What the team has done well

The panel was impressed that:

• The organisation they are part of (NHS Digital) has plans and resources allocated for when services are down.
• The team are using containers so they can replace or add components without stopping or shutting down the service if necessary.

What the team needs to explore

Before their next assessment, the team needs to:

• Test hot-failover plan.
• Test restoring backups

12. Make sure users succeed first time

Decision

The team met point 12 of the Standard.

What the team has done well

The panel was impressed that:

• We saw multiple prototypes that were iterated upon.
• They understand the limitations of the current service and have acted on them.
• The automated tests of current iteration have passed WCAG2.1.
• The team worked within some commercial restrictions (cannot give any competitive advantage to suppliers) but still managed to test with users. In the next phase the commercial restrictions should not be as prevalent, so look forward to more end to end testing showing how suppliers could get on the framework first time.

What the team needs to explore

Before their next assessment, the team needs to:

• Test with more users with diverse accessibility needs.
• Dedicate more time to other elements of the end to end service. We saw the service explored broadly in discovery and alpha but clearly more recent effort has been put into the supplier registration journey. The team needs to make sure they work on the rest of the service.
• Publish a roadmap that relevant stakeholders can engage with.

13. Make the user experience consistent with GOV.UK

Decision

The team met point 13 of the Standard.

What the team has done well

The panel was impressed that:

• The team are using GOV.UK and NHS design patterns and style guides.

What the team needs to explore

Before their next assessment, the team needs to:

• Review that the components they are using are up to date and consistent with current GOV.UK and NHS design systems.

14. Encourage everyone to use the digital service

Decision

The team met point 14 of the Standard.

What the team needs to explore

Before their next assessment, the team needs to:

• Explore further what process could exist if the site is offline.

15. Collect performance data

Decision

The team met point 15 of the Standard.

What the team has done well

The panel was impressed that:

• The team has reviewed lots of data on the current service.
• Privacy concerns were taken into account when choosing an analytics provider.

What the team needs to explore

Before their next assessment, the team needs to:

• Have a more detailed analytics plan for beta before commencing.
• Have the skills in the team to get actionable insights from whichever analytics software they use.

16. Identify performance indicators

Decision

The team met point 16 of the Standard.

What the team has done well

The panel was impressed that:

• The team has a list of KPIs.
• They have benchmarked the current service.

17. Report performance data on the Performance Platform

Decision

The team met point 17 of the Standard.

What the team has done well

The panel was impressed that:

• They are in discussion with the performance platform to report on mandatory KPIs.

18. Test with the minister

Decision

The team met point 18 of the Standard.

What the team has done well

The panel was impressed that:

• They have plans to show the service to senior colleagues in the Department of Health and Social Care and are in contact with relevant stakeholders gathering feedback

What the team needs to explore

Before their next assessment, the team needs to:

• Eventually make plans to test with a minister or relevant senior owner of the service within the NHS.

Conditions before the next assessment

The buying catalogue discovery began in 2017 but this was the teams first assessment. We were shown more than what you would expect in an alpha stage. The team must be more punctual with arranging assessments before they go into public beta.

The team must also submit a spend control form for the next phase (Private Beta) to the Department of Health and Social Care Digital team.

During the assessment the panel clarified that by passing an alpha assessment that would allow them to go into a private beta, not to make the service “live”. The panel would expect the team to restrict access to the service, invite users to use it and get more feedback before the next assessment.

Presenting the Evaluation of digital public health

I’m in my first year on the Digital, Data and Technology Fast Stream (DDaT). This blog post is about my experiences of working in a digital team at Public Health England (PHE).

Joining PHE

Before I joined the fast stream scheme, I graduated with a politics degree from Glasgow University and then worked in the Civil Service as a Visits Officer.

I didn’t have any experience of working in a digital team. You do not need a technical background to join the DDaT Fast Stream. All you need is a passion for making a difference and a desire to be part of the digital transformation in government.

My first placement was as a delivery manager at PHE. As an executive body to the Department of Health and Social Care, PHE has more dealings with operational delivery than setting policy. This set-up arguably positions it closer to the delivery of services than other government departments.

Working on the forefront of PHE’s service delivery offered me a unique insight into the way the organisation uses the power of digital technology to protect and improve the health and wellbeing of the public. It’s a fast-paced agile environment. I had to learn fast about the digital culture of government during my first few months.

On the job

When I joined PHE in October, I was the first fast streamer to be placed in the digital team. This allowed me to explore a wide range of work opportunities and helped me identify areas where I could make a contribution. I got a taste of what digital in government is all about: delivering services for the public.

The brilliant team I joined is the force behind the Evaluation of Digital Public Health. This project is exploring ways to help PHE and the wider health system better demonstrate the impact, cost-effectiveness and benefit of digital health products.

When I joined the team as a delivery manager, the project was at the start of alpha. Over the following 6 months, I witnessed the project evolving from several loose concepts, to wireframes and become a functioning GOV.UK prototype.

Work was often emotional: developing services for the public is both challenging and extremely rewarding. Behind each product is a passionate team, learning to work together alongside academia and other government bodies to solve complex issues. Each team is focused on creating something new and valuable, and as a delivery manager, I relished the challenge of guiding work in this direction.

At the end of March 2019, when I left PHE, the Evaluation of Digital Public Health project was preparing to enter a beta phase.

What I've learned

What makes this fast stream posting different to others is the unique driver behind PHE’s digital products. Unlike traditional teams in government, PHE embeds the science of behaviour change within its services. During the past 6 months, I’ve learned a great deal about creating products that generate positive impact by influencing people’s thoughts and behaviours.

I worked with user researchers and service designers who taught me the importance of delivering services with users in mind. I collaborated with content designers, who emphasised the need for clear, accessible language. I connected with peers from Government Digital Services (GDS), NHS Digital and third sector organisations and I gained an understanding of how to facilitate collaboration with academics to deliver innovative, yet vigorously validated products.

It was an incredible opportunity to dive into the world of public health and the organisations that operate in the sector. What makes PHE stand out is the inspiring and infectious passion of my peers for improving the nation’s health. I’ve built a strong network of colleagues who I’m sure will keep in touch once I move on to my next role.

Find out more about the Evaluation of Digital Public Health project.

Apply for the Digital Data and Technology Fast Stream to start an exciting career in government, transforming technology and building better digital services.

The Digital Weight Management service (currently renamed as: Our Family Health) is a free and fun service that helps the whole family achieve their health goals, one small step at a time. Parents, carers and their children between four and nine years old can use the service any time and anywhere to get trusted, practical and personalised tips, activities and games for the whole family.

Families can set personal goals, track progress, see achievements, and get support from other families to help them stay focused and motivated.

Department / Agency
Public Health England (PHE)

Date of Assessment:
12 February 2019

Assessment Stage:
Alpha

Result of Assessment:
Met

Lead Assessor:
B.Showers

Service Manager:
Jamie Blackshaw

Digital Leader:
Iain O’Neil

Assessment report

Outcome of service assessment

After careful consideration, the panel have decided that the service has met the service standard for Alpha. The panel have the following conditions for the project as it moves into
Private Beta:

• The team should undertake an extended Private Beta that demonstrates the efficacy of the service in improving families’ health. This will involve a follow up period
  sufficient to demonstrate sustained behavioural change.
• The team tests their service with people with low digital capability.

Service assessment

User needs

The team demonstrated a good understanding of the user needs gained through the discovery research and how this integrated with the behavioural analysis. The team were impressed by the great lengths the user researcher had gone to in conducting research with children, immersing themselves in their world to understand their needs. They told the stories of users including parents, children and families and the context of their needs, including the emotional context. This research was focussed on those families who were motivated to change. It would be interesting to explore whether this mirrored those that engaged with the face to face service.

The data from the behavioural analysis research provided support for the data obtained through user research and indicators of how some of the goals the service could be achieved. Research during beta should test whether the service has led to behavioural change and assess whether it is sustained beyond the 12 week programme period.

During assessment the user researcher mentioned user research with access needs however it was not clear whether the wider team fully understood what this involved. Whilst the team have built accessibility testing and accessibility research into their beta plan, it would be valuable for the team as a whole to understand what face to face research with users with access needs can bring over and above adherence to WCAG guidelines and expert testing centres (e.g. DAC).

It would also be valuable for the team to test their service with people with low digital ability. The team stated that they did not have any users with low digital ability that would use their service so they had not and did not intend to test with people with them. They also stated that according to their data between 1-3% of their users would have low digital ability. That shows they have identified users with low digital ability and that group consists of up to 3% of a large sample of the population. Furthermore, testing with these groups provides insight that enables a better experience for all users and one that is inclusive.

Team

The panel were impressed by the overall team and with the relationship between the supplier and the core PHE team. The PHE policy lead was the service owner for the activity and was both engaged and empowered in his everyday work with the project.

There was a clear separation between the design aspects and the role of the user researcher and this should continue into Beta. Overall the balance of roles was appropriate for the alpha and the focus on service design that the project has and the panel were impressed by the multidisciplinary team that included obesity and healthy weight expertise, behavioural insights and design thinking.

The project was also able to articulate how the team will evolve as it moves into Beta.

The Alpha team included:

• Service owner (PHE)
• Service manager (PHE)
• Behavioural insights lead (PHE)
• Digital consultant (PHE)
• Design director
• Delivery manager
• Interaction designer
• User researcher
• Service designer
• Technical lead

The team operated both as a co-located team as well as employing ways of working that included weeknotes, show and tells and collaborations tools such as Jira and Realtime.

The team were a mix of PHE and an external supplier. In Alpha the balance of roles was toward the supplier side, but the team described how this will shift towards PHE providing more resource as the project moves through Beta.

Technology

Due to the complexity of the problem the team are trying to solve the team have elected focus on the value proposition for users, testing with wire frames rather than coded prototypes.

The team have made a series of recommendations which will need to be explored before entering private beta. The proposed solution is a responsive web app this is a suitable choice for this use case given the core user group and the team have acknowledged the potential issues around access this may cause.

The proposed tech stack was developed in conjunction with PHE’s digital team, based on the existing infrastructure in PHE and open source technologies. The recommendations include advice on best practice for open sourcing, security and resilience.

The panel recommends the team conduct a technical review with the PHE digital team and DHSC Digital once they have a working build of the service.

Design

The focus of the Alpha was primarily on understanding the value proposition for a family-centric digital weight management service and testing iterations with users.

The team could show how the design of the service had been iterated based on user research including the choice of brand and the need for alternative content to that currently available. It would be valuable for the team to consider how they could work with current partners in the NHS to understand the wider service design and explore the possibility of shared content development to avoid duplication and support a better experience for users. It would also be valuable for the team to ensure that where user research suggests design decisions that deviate from current patterns and standards, this evidence and decisions are fed back to the relevant standards team.

Analytics

As part of Alpha, the team have been doing some good thinking around KPIs and success measures. These included Theory of Change and building on best practice being developed as part of another PHE project exploring the evaluation of digital services.

In Beta the team should be clear about how they’re going to collect the data to measure against the KPIs.

Recommendations

In addition to the conditions at the start of this report, the panel also have a few recommendations it would like the project to consider as it moves into beta:

1. As the project moves into Beta consideration should be given to the sustainability of aspects such as curation of content and resources. How will this work long-term and what would the impact be on the service in BAU? Are there any other products or services that they could work with (e.g. work with NHS UK to redevelop their content).
2. Consideration should be given to how does this app fit into the other user journeys e.g. adult weight and other health related apps?