IES Trials Unit: reflections as we embrace year three

Blog posts

16 Sep 2024

Dr Susie Bamford, Principal Research Fellow
Dr Alexandra Nancarrow, Research Fellow

E is for evidence: empowering employment and education policy decisions through robust evaluations

The Trials Unit at the Institute for Employment Studies (IES) has the goal of providing evidence for policy and practice in the education and employment arenas. There’s a plethora of ideas and interventions intended to improve practice, but how do we know what works? What is worth pursuing and implementing? What will make a difference?

The Trials Unit aims to answer these questions by carefully designing robust evaluations to make meaningful comparisons between groups and outcomes and draw evidence-based conclusions. Our gold standard measure is the randomised controlled trial (RCT) and within that we use both quantitative methods, to measure impact, and a mixture of qualitative and quantitative methods to examine the processes behind how the interventions might work. We consider and implement quasi-experimental designs (QEDs) when RCTs may not be feasible or an ethical approach in the context of the intervention. And as well as using a range of counterfactual impact designs, we also support the incubation of interventions through Theory of Change development and pilots to understand key elements of their design and the effects these have. These projects can be precursors to trial and QED studies.

Our interests span a variety of fields including education, employment, and health. This is reflected in our policy research for the main government departments and what works centres, as well as studies within employers on workforce practices.

Key achievements to date

Our trials work has started strongly with several exciting projects, and here we note some highlights.

We completed a trial of the Flexible Phonics programme which extends usual phonics learning methods by adding an extra step. This Education Endowment Foundation (EEF) funded RCT examined the programme’s impact on children’s reading of words. Recently, we published an updated report on the trial outcomes which includes a further exploration of the data for particular subgroups. This provided some evidence that there was a benefit for children in schools who also completed the Nuffield Early Language Intervention. However, overall, pupils who participated in Flexible Phonics made the equivalent of one month less progress, on average, in early word recognition than pupils who did not receive the programme. There are several possible reasons for this interesting finding, and you can read more in the report.

A highlight for us has been the Health-led Trials (HLT) evaluation, which was our RCT that examined Individual Placement and Support (IPS) for individuals in primary care with moderate to mild physical and mental health conditions. This was the largest trial of IPS that has been seen internationally - recruiting over 9,700 people including a group in work who were struggling as well as those inactive or unemployed. The trial results were intriguing with different impacts (employment vs health) seen between the two delivery sites. The comprehensive evaluation data meant it was possible to dig into the causes of this. Read more about this in Becci Newton’s blog here and find the reports here.

We were also excited to work with a FTSE 100 employer to support on the development, trialling and evaluation of a workplace health and wellbeing intervention in sites across Europe, Asia, and the Americas. This project is commercially sensitive so we can’t say too much about it here, but it is a fascinating piece of work in our portfolio.

Additionally, we completed an efficacy trial of the Tips by Text programme. This is an initiative which comprises text messages to parents and aims to improve their 4–5-year olds’ language, literacy, numeracy, and social and emotional skills. This trial occurred during the pandemic, so attrition was high, but we were still able to continue the evaluation. This found the intervention had no impact on the outcome measures, but results should be treated with caution. We are continuing this work currently with an effectiveness trial, under more usual non-pandemic circumstances!

More information regarding the Trials Unit’s early work can be found in a news article published by IES in April 2023.

Current projects

We are now into our third year of the IES Trials Unit, and we continue to design quality evaluations on a variety of topics: from adult learning and its links with employment opportunities, back to the beginnings of the education journey in the early years, to explore what works at closing the disadvantage gap at this stage.

We are delighted to currently be working on the following projects:

Multiply – evaluating a suite of adult maths interventions
Funded by the Department for Education (DfE), IES is part of a notable consortium of expert evaluators who have been selected to deliver a suite of trials to evaluate several adult numeracy interventions. Working closely with the intervention delivery teams, we have been designing trials and considering QED and pilot alternatives to evaluate individual interventions. IES is currently leading the evaluation work on three interventions, which include two adult maths course-based interventions and a maths engagement programme for families.

Early years – reducing the disadvantage gap
Many of our triallists have specialised expertise in education and psychology and a strong interest in finding interventions that improve early years outcomes, especially for disadvantaged children. We are currently running two early years evaluations funded by the EEF; Tips by Text, and Tales Toolkit.

IES has partnered with the National Institute of Economic and Social Research (NIESR) to deliver the effectiveness trial of the Tips by Text programme. The trial will go live in schools in September. This effectiveness trial will build on the evidence we gathered as part of the efficacy trial which you can read about here.

We are also continuing to conduct a pilot study for Tales Toolkit. This is a package of training and resources supporting 3 – 5-year-old children’s early language and social development through oral storytelling. The study is exploring how Tales Toolkit is delivered in practice in thirty early years settings, the feasibility of delivering the programme, and early years practitioners’ experiences of using the programme. IES is gathering evidence from surveys, observations, a review of the materials, and interviews with stakeholders. This will generate valuable insights into whether the intervention is trial ready.

Alternative methodology evaluations

To complement our RCT work, we have some exciting evaluations utilising other robust methodologies. In the Youth Futures Foundation (YFF) space, we are incubating trials on several of our current projects: one to understand if a youth employment initiative could be scaled to trial, and another to explore if employment schemes for young people in the justice system would be trialable. Based on our feasibility work, both appear promising.

For the Department for Education (DfE), we are working with NIESR on an evaluation of the Department’s language programmes. Using a matching QED design, we are leading the impact evaluation, examining impact on the attainment and employment of English Language Assistants.

We are excited to continue excellent evaluation work to provide key evidence for policy in the employment and education spaces.

Team and expertise

Anneka Dawson DPhil is the head of the Trials Unit. While she is on maternity leave until November 2024, the Trials Unit is being steered by Dr Susie Bamford, Dr Allie Nancarrow, and Dr Seemanti Ghosh, with leadership oversight from IES Director Becci Newton. Members of the IES Trials Unit team include experts across trial design and methodology, data management, qualitative research, statistics, education, economics, and psychology, many of whom are also accredited SRS researchers.

Future directions

We are passionate about designing and delivering high quality evaluations to help discover what works. If you are interested in learning more about the IES Trials Unit and/or how we could work with your organisation, please contact Dr Susie Bamford or Dr Allie Nancarrow.

Subscribe to blog posts

Any views expressed are those of the author and not necessarily those of the Institute as a whole.