Background

In our pre-COVID report to Council Scrutiny Board in February 2020, we set out our approach to evaluating our work and measuring Return on Investment:

It is vital to measure place-based outcomes in order to fully evaluate the impact of the 100% Digital Leeds programme.

Digital inclusion is a means to an end, it is not an end in itself. Digital inclusion is not about digital: it is about inclusion.

People can use the internet to tackle many of the challenges and inequalities they face on a daily basis.

Being online means having access to cheaper goods, services and utilities, more employment opportunities, self-management tools for long-term health conditions and easier ways to deal with the council and government departments.

Measurement framework

In our 2019 report to Scrutiny Board, we discussed our plans to create a sustainable evaluation framework to measure the programme’s Return on Investment, social impact and progress towards 100% digital inclusion.

Over the course of the year the digital inclusion team have worked with Good Things Foundation to design this model, drawing on that organisation’s expertise as a national leader in the research and evaluation of digital inclusion programmes, including work with Government and other national stakeholders such as: Department for Education, NHS England/ NHS Digital, Ministry of Housing, Communities and Local Government, Centre for Ageing Better, BT, Lloyds Banking Group.

We now have an evaluation framework that enables us to measure improved outcomes across a range of indicators. Our evaluation framework also gives us a methodology that we can use to report the return on investment that digital inclusion brings to residents, the council and the city as a whole.

Evaluation framework

The positive social impact we wish to measure includes:

• Progression to further learning (both formal and informal)
• Progression to positive employment outcomes (both in work and into work)
• Improvements in health and wellbeing (from using digital health tools to self-manage conditions)
• Increased social connection and reduced isolation

The framework includes:

• Ongoing User Progression Survey
  This collects demographic data to measure trends and to build a profile of end users (e.g. age, employment status, if they have a disability/long term health condition, language needs, whether they’re a Housing Leeds tenant). The survey also includes attitudinal and behavioural change, and calculation of channel shift savings. This will be rolled out to all of the 100% Digital Leeds partners who offer digital access and support to users
• Monthly Activity Survey for organisations participating in the tablet lending scheme
• Monthly Activity Survey for organisations benefiting from Leeds Digital Inclusion Funding stream
  Both of these are designed to collect data to provide quantitative evidence of impact delivered by partners in communities and among target audiences, so that we can state, for example, “x number of organisations in the movement are helping people find employment and saw y amount of people this quarter.”

  The digital inclusion team will also visit each organisation quarterly to collect qualitative evidence in the form of user case studies and organisational case studies, featuring quotes and images. These will be organised in categories relevant to specific agendas e.g. health, employment and skills, financial resilience, community integration, reduced isolation or loneliness and greater independence.

• Ongoing Digital Champions progression survey
  As well as continuing to evaluate the effectiveness of Digital Champions training, the impact of the practical application of training and the engagement of end users by Digital Champions will also be measured and reported.

In addition to these surveys for the more engaged organisations that are part of the 100% Digital Leeds network, the digital inclusion team will continue to maintain quarterly contact with less engaged organisations in the network to identify any support they might need to boost their progress and engagement with end users.

This data will enable us to report:

• Number of overall organisations, Online Centres, tablet lending participants, Digital Champions, Learn My Way users in the movement, demonstrating the reach and visibility of the programme
• Nature of the digital inclusion activity of organisations and Digital Champions
• Number of end users benefiting plus their demographic data
• Profile of individuals benefiting from the programme in the form of case studies
• Profile of organisations benefiting from the programme in the form of case studies
• Channel shift savings to public services – Leeds City Council, NHS, JobCentre Plus, other local/national government departments

Results from the User Progression Survey enable us to produce an estimate of the value of channel shift as a result of Leeds residents gaining digital skills.

As people move to online transactions to replace phone calls and visits, cost savings can be applied to those behavioural changes. This is an important aspect of the continuing evaluation of the 100% Digital Leeds programme.

Annualised savings

A worked example of annualised savings as a result of behaviour changes and channel shift due to the 100% Digital Leeds programme is below:

It should be noted that this framework was developed and used pre-COVID and some elements have not been used during the pandemic. We are reviewing and adapting the framework and we also use other evaluation tools and methodologies for specific strands of work.