Impact Workstream Round Table Event

Measuring the Impact of Modern Apprenticeships

Introduction

On the afternoon of 18 August 2017 the Centre for Work Based Learning held a round table event on measuring the impact of modern apprenticeships (MAs), which is part of the Centre’s Impact workstream. The aim of the roundtable was to allow attendees to contribute critically and constructively to the approach that has been taken, and will be taken, in the evaluation of MAs. Stephen Boyle, who is the Chief Economist at the Royal Bank of Scotland, chaired the event. Attendees, with expertise in a particular area related to measuring the impact of learning, included representatives from the Scottish Government, Scottish Funding Council, University of Strathclyde and University of Westminster. Below are some of the key points and highlights from the discussion.

 

Work completed so far

First of all, attendees at the roundtable were given a history of MA research and evaluation activity and an indication of research gaps.

Measuring the impact of MAs has done for the past 6 years, meaning we currently possess information on the views of stakeholders, benefits of the programme and short term outcomes to individuals and employers.

In 2014, Audit Scotland examined the MA programme and recommended that the long term outcomes of MAs were investigated. SDS then worked with the OECD to develop a framework for evaluating these long term outcomes, which was published in December 2016. The framework components are shown below. 

Aims of the MA Programme

Attendees were clear that the aims of the MA programme must be transparent as this will guide the evaluation process. The two main policy objectives of the programme were stated to be; a method to tackle youth unemployment and to develop the skills of the workforce. However, there was also an acknowledgement that the MA programme is demand-led, driven by the needs of employers.  

 

Counterfactual and Comparisons

Obtaining the correct counterfactual to be used as a comparison in assessing the impact of MAs on individuals is of paramount importance, yet one of the main methodological issues of the evaluation [1]. Although obtaining the correct counterfactual group is a challenge that will need to be overcome, most of the discussion at the roundtable focused on assessing the impact of the programme by individual characteristics, MA framework and by different learning and training routes.

Although being able to compare the impact of the MA programme against the impact of other forms of post-16 learning would be a “massive step in data” (as stated by one participant), it was mentioned that the analysis in place and proposed for the evaluation of the MA programme is...

“...exactly the kind of analysis we should be applying across the whole spectrum of skills provisions”.

Dominic Munro, Director for Fair Work, Employability and Skills, Scottish Government

Data linkage

The framework for evaluating long term outcomes of MAs, developed by the OECD, states that long term outcomes are best measured through the linking of administrative datasets. Although the high quality of the work conducted by OECD was highlighted, attendees were also aware that this framework is the ‘gold standard’ in data linkage and analysis which will take time to ensure the legal framework is in place for the sharing of data and the development of the dataset.

Yet attendees stated that they are keen to be realistic and practical about what information can be offered by the data and when. This discussion led on to what we can do now while we wait for data linkage to be complete – something that attendees stated as important to do.

 

Importance of measuring ‘soft’ outcomes

Although not the direct focus of this roundtable, attendees were keen to highlight the importance of measuring the wider, softer benefits of the MA programme. However, measuring these outcomes can be considered to be more difficult than measuring the ‘hard’ economic impacts of the MA programme. The upcoming MA Wellbeing Survey will pilot an approach to measuring the subjective wellbeing and wider outcomes of the programme on individuals.

"We need to measure what is important, not what is easy to measure

Professor Peter McGregor, Fraser of Allander Institute

Benefits to employers

The employers gave the view that it was easy for them to see the impact that MAs have on their business. However, this also led to a discussion about whether the benefit to employers and the individual is due to unobserved characteristics of the individuals or the MA programme. Further work is needed in this area.

 

Uses of the analysis

The group was in agreement that conducting an evaluation into the impacts of the MA programme was a worthwhile endeavour, despite the many challenges in doing so. To this end, attendees were keen to discuss the potential uses of the analysis once it has been completed. The analysis appears to have multiple uses;

  • A performance measure to check that the programme is meeting its aims;
  • To be used in Careers Information Advice and Guidance to assist in individuals’ decision making;
  • To examine returns to the apprenticeship system
  • To be used in policy decision making and to examine where to focus investment.

Although the information produced in the evaluation of the MA programme could have possible uses in informing decision making of both individuals and policy makers, attendees also agreed that other factors were at play. There is also the recognition that we need to further develop our understanding of how young people make career decisions in order to use any analysis to effectively inform such choices.

Next steps

Overall, the discussions at the round table provided a beneficial contribution to the past, present and future evaluations of MAs. The insights and ideas generated from the discussion will be used to inform the initial findings and next steps to be taken in the Impact work stream’s research projects.

 

[1] See OECD Framework to further discussions around a suitable control group

The Round Table brought together some of the foremost experts in Scottish economic affairs, along with senior figures from Government and industry.  The group reaffirmed the importance of work based learning to the future prosperity of our country and agreed the approach which should be taken to fully evaluating the economic and social benefits of Apprenticeships.  This important event has done vital ground work which will shape our research activity over the coming months.

David Coyne, Centre for Work-based Learning

This site uses cookies and other tracking technology to assist with navigation and your ability to provide feedback, analyse your use of our products and services, assist with our promotional efforts, and provide content from third parties. Review Cookie Policy