On the afternoon of 18 August 2017 the Centre for Work Based Learning held a round table event on measuring the impact of modern apprenticeships (MAs), which is part of the Centre’s Impact workstream. The aim of the roundtable was to allow attendees to contribute critically and constructively to the approach that has been taken, and will be taken, in the evaluation of MAs. Stephen Boyle, who is the Chief Economist at the Royal Bank of Scotland, chaired the event. Attendees, with expertise in a particular area related to measuring the impact of learning, included representatives from the Scottish Government, Scottish Funding Council, University of Strathclyde and University of Westminster. Below are some of the key points and highlights from the discussion.
First of all, attendees at the roundtable were given a history of MA research and evaluation activity and an indication of research gaps.
Measuring the impact of MAs has done for the past 6 years, meaning we currently possess information on the views of stakeholders, benefits of the programme and short term outcomes to individuals and employers.
In 2014, Audit Scotland examined the MA programme and recommended that the long term outcomes of MAs were investigated. SDS then worked with the OECD to develop a framework for evaluating these long term outcomes, which was published in December 2016. The framework components are shown below.
Attendees were clear that the aims of the MA programme must be transparent as this will guide the evaluation process. The two main policy objectives of the programme were stated to be; a method to tackle youth unemployment and to develop the skills of the workforce. However, there was also an acknowledgement that the MA programme is demand-led, driven by the needs of employers.
Obtaining the correct counterfactual to be used as a comparison in assessing the impact of MAs on individuals is of paramount importance, yet one of the main methodological issues of the evaluation [1]. Although obtaining the correct counterfactual group is a challenge that will need to be overcome, most of the discussion at the roundtable focused on assessing the impact of the programme by individual characteristics, MA framework and by different learning and training routes.
Although being able to compare the impact of the MA programme against the impact of other forms of post-16 learning would be a “massive step in data” (as stated by one participant), it was mentioned that the analysis in place and proposed for the evaluation of the MA programme is...
Dominic Munro, Director for Fair Work, Employability and Skills, Scottish Government
The framework for evaluating long term outcomes of MAs, developed by the OECD, states that long term outcomes are best measured through the linking of administrative datasets. Although the high quality of the work conducted by OECD was highlighted, attendees were also aware that this framework is the ‘gold standard’ in data linkage and analysis which will take time to ensure the legal framework is in place for the sharing of data and the development of the dataset.
Yet attendees stated that they are keen to be realistic and practical about what information can be offered by the data and when. This discussion led on to what we can do now while we wait for data linkage to be complete – something that attendees stated as important to do.
Although not the direct focus of this roundtable, attendees were keen to highlight the importance of measuring the wider, softer benefits of the MA programme. However, measuring these outcomes can be considered to be more difficult than measuring the ‘hard’ economic impacts of the MA programme. The upcoming MA Wellbeing Survey will pilot an approach to measuring the subjective wellbeing and wider outcomes of the programme on individuals.
Professor Peter McGregor, Fraser of Allander Institute
The employers gave the view that it was easy for them to see the impact that MAs have on their business. However, this also led to a discussion about whether the benefit to employers and the individual is due to unobserved characteristics of the individuals or the MA programme. Further work is needed in this area.
The group was in agreement that conducting an evaluation into the impacts of the MA programme was a worthwhile endeavour, despite the many challenges in doing so. To this end, attendees were keen to discuss the potential uses of the analysis once it has been completed. The analysis appears to have multiple uses;
Although the information produced in the evaluation of the MA programme could have possible uses in informing decision making of both individuals and policy makers, attendees also agreed that other factors were at play. There is also the recognition that we need to further develop our understanding of how young people make career decisions in order to use any analysis to effectively inform such choices.
Overall, the discussions at the round table provided a beneficial contribution to the past, present and future evaluations of MAs. The insights and ideas generated from the discussion will be used to inform the initial findings and next steps to be taken in the Impact work stream’s research projects.
[1] See OECD Framework to further discussions around a suitable control group
David Coyne, Centre for Work-based Learning
