Launch of the IES Trials Unit: an explainer

Blog posts

26 Apr 2022

Anneka Dawson, Head of IES Trials Unit and Pre-16 education

As we launch our IES Trials Unit, Anneka Dawson details what we mean by trials and how they can add an essential evidence base for making important business decisions.

What do you mean by trials?

Randomised controlled trials (RCTs) are considered the gold standard in evaluation social science research, helping establish causality where it has been lacking. We are proud to work with six of the What Works centres and affiliates, who are independent organisations tasked with helping ensure that public spending goes to interventions and projects that will help improve things for societies greatest challenges. However, our work with them has shown that not all projects are suitable for RCTs and quasi-experimental designs are sometimes more realistic or palatable to different target groups. Working in social science research means we have to consider what is feasible, but also ensuring the research is as robust as possible. There are lots of options outside of RCTs for us to consider, and research is starting to demonstrate that they can be viable alternatives. Therefore, we cover both RCTs and quasi- experimental designs and can also provide advice on suitable methods for the project in question.

Does IES have experience in trials?

The Institute has been working on trials (our broad interpretation of trials includes RCTs and quasi- experimental designs such as matching techniques) for many years and across a diverse spectrum of disciplines. These cover the areas you would expect from IES, in how to improve employment outcomes for clients such as the Department for Work and Pensions and projects for specific employers such as RSSB, testing out mental health training for managers.

Why now?

We have been building and developing internal expertise across these projects and the next stage of our strategic plan is to ensure the wider world knows about what we have been doing and what we can offer to reach new partners and clients.

How can trials assist with my objectives?

Robust research trials build the evidence base for making crucial decisions for businesses, schools, local authorities and government departments. For example, our recent RCT on a parenting intervention called Tips By Text, for the Education Endowment Foundation will produce a ‘month’s progress’ score for the programme so that schools can decide if it is a worthwhile intervention to dedicate school budget and time to.  A further example is our research for RSSB that showed face-to-face training and e-learning were of equal value in helping to train managers. This helps RSSB decide which method to implement as they are both equally effective; so the decision can be based on other factors such as individual preference or cost of delivery.

What are the ingredients of a successful trial?

1. A clear counterfactual group (or ‘business as usual’ group who have continued their usual practice) to compare to a group who have received an intervention, so we can see what would have happened without that intervention.
2. A carefully constructed plan or protocol which details in advance what is being tested in the trial, with outcome measures identified, a clear methodology and timeline. Ideally this would be published on a trial registry and any changes to this plan clearly identified and explained with an update, so that we are transparent in what we are doing and why.
3. A detailed statistical analysis plan which outlines what analysis will be undertaken in advance- this should also be published.
4. A Theory of Change or logic model should be used to develop the protocol and statistical analysis plan. If one does not exist for an intervention already, we will work with a client or delivery team to construct one using a series of Theory of Change workshops.
5. Impact evaluation of outcomes and a complementary Implementation and Process Evaluation (IPE) are equally important elements of a trial so that we know what has changed and the size of that impact (through the impact evaluation) as well as exploring why something might have changed and the mechanisms of change through the IPE.

Some projects may be investigating a proof of concept for a trial so would concentrate on developing a theory of change and investigating these areas for a future trial.

Do you have any specific examples?

Aside from the projects mentioned above regarding employment outcomes, the projects IES are working on also cover topics as wide ranging as the impacts of mindfulness practice,  phonics teaching in schools, supporting people with disabilities and health conditions, texting nudges to parents to help support their children’s literacy and numeracy development and training for early years practitioners to help STEM skills development in two year olds.

What next?

We are keen to assist with any potential requirements. We have teams set up to help at each step of the way from project design, management and planning through all the stages of delivery, primary and secondary data collection, analysis and report writing to the highest level.

Please get in contact if you have a project in mind!

Subscribe to blog posts

Any views expressed are those of the author and not necessarily those of the Institute as a whole.