Skip to content
Digital Inclusion Toolkit Logo
  • Home
  • >
  • Developing a digital inclusion programme
  • >
  • Evaluating a digital inclusion project
Read Discussion (3 )

  • Background
  • Measurement framework
  • Annualised savings

Evaluating a digital inclusion project

Last updated: December 10, 2020 by Leeds City Council

Funders will want to know the impact of their investment, and you’ll want to know what’s worked well and not so well. Here’s the Leeds experience, and we’re keen to hear your feedback.

Background

In our pre-COVID report to Council Scrutiny Board in February 2020, we set out our approach to evaluating our work and measuring Return on Investment:

It is vital to measure place-based outcomes in order to fully evaluate the impact of the 100% Digital Leeds programme.

Digital inclusion is a means to an end, it is not an end in itself. Digital inclusion is not about digital: it is about inclusion.

People can use the internet to tackle many of the challenges and inequalities they face on a daily basis.

Being online means having access to cheaper goods, services and utilities, more employment opportunities, self-management tools for long-term health conditions and easier ways to deal with the council and government departments.

Measurement framework

In our 2019 report to Scrutiny Board, we discussed our plans to create a sustainable evaluation framework to measure the programme’s Return on Investment, social impact and progress towards 100% digital inclusion.

Over the course of the year the digital inclusion team have worked with Good Things Foundation to design this model, drawing on that organisation’s expertise as a national leader in the research and evaluation of digital inclusion programmes, including work with Government and other national stakeholders such as: Department for Education, NHS England/ NHS Digital, Ministry of Housing, Communities and Local Government, Centre for Ageing Better, BT, Lloyds Banking Group.

We now have an evaluation framework that enables us to measure improved outcomes across a range of indicators. Our evaluation framework also gives us a methodology that we can use to report the return on investment that digital inclusion brings to residents, the council and the city as a whole.

Evaluation framework

The positive social impact we wish to measure includes:

  • Progression to further learning (both formal and informal)
  • Progression to positive employment outcomes (both in work and into work)
  • Improvements in health and wellbeing (from using digital health tools to self-manage conditions)
  • Increased social connection and reduced isolation

The framework includes:

  • Ongoing User Progression Survey
    This collects demographic data to measure trends and to build a profile of end users (e.g. age, employment status, if they have a disability/long term health condition, language needs, whether they’re a Housing Leeds tenant). The survey also includes attitudinal and behavioural change, and calculation of channel shift savings. This will be rolled out to all of the 100% Digital Leeds partners who offer digital access and support to users
  • Monthly Activity Survey for organisations participating in the tablet lending scheme
  • Monthly Activity Survey for organisations benefiting from Leeds Digital Inclusion Funding stream
    Both of these are designed to collect data to provide quantitative evidence of impact delivered by partners in communities and among target audiences, so that we can state, for example, “x number of organisations in the movement are helping people find employment and saw y amount of people this quarter.”

    The digital inclusion team will also visit each organisation quarterly to collect qualitative evidence in the form of user case studies and organisational case studies, featuring quotes and images. These will be organised in categories relevant to specific agendas e.g. health, employment and skills, financial resilience, community integration, reduced isolation or loneliness and greater independence.

  • Ongoing Digital Champions progression survey
    As well as continuing to evaluate the effectiveness of Digital Champions training, the impact of the practical application of training and the engagement of end users by Digital Champions will also be measured and reported.

In addition to these surveys for the more engaged organisations that are part of the 100% Digital Leeds network, the digital inclusion team will continue to maintain quarterly contact with less engaged organisations in the network to identify any support they might need to boost their progress and engagement with end users.

This data will enable us to report:

  • Number of overall organisations, Online Centres, tablet lending participants, Digital Champions, Learn My Way users in the movement, demonstrating the reach and visibility of the programme
  • Nature of the digital inclusion activity of organisations and Digital Champions
  • Number of end users benefiting plus their demographic data
  • Profile of individuals benefiting from the programme in the form of case studies
  • Profile of organisations benefiting from the programme in the form of case studies
  • Channel shift savings to public services – Leeds City Council, NHS, JobCentre Plus, other local/national government departments

Results from the User Progression Survey enable us to produce an estimate of the value of channel shift as a result of Leeds residents gaining digital skills.

As people move to online transactions to replace phone calls and visits, cost savings can be applied to those behavioural changes. This is an important aspect of the continuing evaluation of the 100% Digital Leeds programme.

Annualised savings

A worked example of annualised savings as a result of behaviour changes and channel shift due to the 100% Digital Leeds programme is below:

Service

Savings

A&E £310,426
GP £230,427
Leeds City Council £98,686
Other government offices £72,703
Jobcentre Plus £69,370
Total £781,612

It should be noted that this framework was developed and used pre-COVID and some elements have not been used during the pandemic. We are reviewing and adapting the framework and we also use other evaluation tools and methodologies for specific strands of work.

Contribute to this chapter

If you have suggestions please comment in the page discussion.

3 comments

3 comments on “Evaluating a digital inclusion project”

  1. Avatar Sam White says:
    December 11, 2020 at 12:10 pm

    Hi Leeds and everyone,

    How did you calculate your annualised savings? Was your methodology something you can share or would it be too specific to your own processes to really be viable for others to use?

    Reply
  2. Leeds City Council Leeds City Council says:
    December 11, 2020 at 6:31 pm

    Hi Sam. This was something we worked on in partnership with Good Things Foundation. Some of the calculations and sources for analysis are similar to those that Good Things Foundation used in their Blueprint for a 100% Digitally Included UK: https://www.goodthingsfoundation.org/sites/default/files/blueprint-for-a-100-digitally-included-uk-0.pdf

    As it says in the article, we’re going to review our previous methodologies and the financial modelling we used in light of COVID. We’ll post about that here as the work gets underway.

    Reply
  3. Avatar Jane Goodison says:
    December 16, 2020 at 3:14 pm

    Do you have a suggested form template for gathering this evidence that could be shared?

    Reply

Discussion - “Evaluating a digital inclusion project” Cancel reply

Your email address will not be published. Required fields are marked *

Your input is important to us

digitalinclusionkit.org is founded on knowledge built through collaboration of local authorities.

If you would like to contribute please join in the discussion or send us a message.

Digital Inclusion Toolkit Logo
  • About us
  • Contact us
  • Accessibility
  • Privacy
  • Cookies
How we use cookies
We would like to use cookies to help give the best online experience. Please let us know if you agree to this.
Cookie settingsAccept and close
Manage consent

Cookies

This website uses cookies to assist in core services to support your use of our website.
Necessary cookies
Always Enabled

Cookies are files saved on your phone, tablet or computer when you visit a website. We use this information to make the website work as well as possible and improve digitalinclusionkit.org. To find out more about controlling and deleting cookies usage on your browser, visit www.aboutcookies.org.

Non-necessary cookies

This site uses Google Analytics to track how the website is used by recording clicks on links and information about the device used to browse the site. This site uses HotJar to record how the website is used and to collect information from users in polls or surveys.