Background

In our pre-COVID report to the Scrutiny Board in February 2020, we set out our approach to evaluating our work and measuring return on investment. There were four elements that drove our approach. We stressed that:

1. it is vital to measure place-based outcomes in order to fully evaluate the impact of the 100% Digital Leeds programme
2. digital inclusion is a means to an end, it is not an end in itself. Digital inclusion is not about digital, it is about inclusion
3. people can use the internet to tackle many of the challenges and inequalities they face on a daily basis
4. being online means having access to cheaper goods, services and utilities, more employment opportunities, self-management tools for long-term health conditions and easier ways to deal with the council and government departments

Creating a sustainable evaluation framework

In our 2019 report to the Scrutiny Board, we discussed our plans to create a sustainable evaluation framework to measure:

• the programme’s return on investment
• social impact
• progress towards 100% digital inclusion

Over the course of the year the digital inclusion team has worked with Good Things Foundation to design this model. We drew on their expertise as a national leader in the research and evaluation of digital inclusion programmes.

This includes work with government and other national stakeholders such as:

• Department for Education
• NHS England and NHS Digital
• Ministry of Housing, Communities and Local Government
• Centre for Ageing Better
• BT
• Lloyds Banking Group

We now have an evaluation framework that allows us to measure improved outcomes across a range of indicators. Our evaluation framework allows us to report on the return on investment that digital inclusion brings to:

• residents
• the council
• the city as a whole

Measuring positive social impact

The positive social impact we want to measure includes:

• progression to further learning (both formal and informal)
• progression to positive employment outcomes (both in work and into work)
• improvements in health and wellbeing (from using digital health tools to self-manage conditions)
• increased social connection and reduced isolation

Evaluation using surveys

The framework includes a number of surveys. We used these to evaluate a range of outcomes.

Ongoing user progression survey

This survey collects demographic data to measure trends. It builds a profile of end users. It records their:

• age
• employment status
• disabilities or long term health conditions
  language needs
• housing status (whether they are a Housing Leeds tenant)

The survey also records attitudinal and behavioural change, and calculates channel shift savings. This survey will be rolled out to all of the 100% Digital Leeds partners who offer digital access and support to users.

Monthly activity surveys

We run two monthly activity surveys, for organisations participating in the tablet lending scheme and organisations benefiting from the Leeds Digital inclusion funding stream.

Both of these surveys are designed to collect data to provide quantitative evidence of impact delivered by partners in communities and among target audiences. This is so that we can state, for example, “x number of organisations in the movement are helping people find employment and saw y amount of people this quarter.”

The digital inclusion team visits each organisation every three months to collect qualitative evidence. This is in the form of user case studies and organisational case studies, featuring quotes and images.

These are organised in categories relevant to specific agendas. For example:

• health, employment and skills
• financial resilience
• community integration
• reduced isolation or loneliness
• greater independence

Ongoing Digital Champions progression survey

As well as continuing to evaluate the effectiveness of Digital Champions training, we also measure and report on the impact of the practical application of training and the engagement of end users.

Maintaining regular contact with network organisations

In addition to these surveys for the more engaged organisations that are part of the 100% Digital Leeds network, the digital inclusion team maintains quarterly contact with less engaged organisations in the network. This is to identify any support they might need to boost their progress and engagement with end users.

Data reporting

This data allows us to report on:

• the reach and visibility of the programme, by reporting the number of overall organisations, online centres, tablet lending participants, Digital Champions and Learn My Way
• the nature of the digital inclusion activity of organisations and Digital Champions
• the number of end users and their demographic data
• the profile of people benefiting from the programme, in the form of case studies
• the profile of organisations benefiting from the programme, in the form of case studies
• channel shift savings to public services – Leeds City Council, NHS, JobCentre Plus and other local and national government departments

Results from the user progression survey allow us to produce an estimate of the value of channel shift as a result of Leeds residents gaining digital skills.

As people move to online transactions to replace phone calls and visits, cost savings can be applied to those behavioural changes. This is an important aspect of the continuing evaluation of the 100% Digital Leeds programme.

Annualised savings

We have detailed an example below of annualised savings. They were achieved as a result of behaviour changes and channel shift due to the 100% Digital Leeds programme.

It should be noted that this framework was developed and used pre-COVID and some elements have not been used during the pandemic. We are reviewing and adapting the framework and we also use other evaluation tools and methodologies for specific strands of work.