Uncovering individualised treatment effects for educational trials

Abstract

Large-scale Randomised Controlled Trials (RCTs) are widely regarded as “the gold standard” for testing the causal effects of school-based interventions. RCTs typically present the statistical significance of the average treatment effect (ATE), which captures the effect an intervention has had on average for a given population. However, key decisions in child health and education are often about individuals who may be very different from those averages. One way to identify heterogeneous treatment effects across different individuals, not captured by the ATE, is to conduct subgroup analyses. For example, free school meal (FSM) pupils as required for projects funded by the Education Endowment Foundation (EEF) in England. These subgroup analyses, as we demonstrate in 48 EEF-funded RCTs involving over 200,000 students, are usually not standardised across studies and offer flexible degrees of freedom to researchers, potentially leading to mixed, if not misleading, results. Here, we develop and deploy an alternative to ATE and subgroup analysis, a machine-learning and regression-based framework to predict individualised treatment effects (ITEs). ITEs could show where an intervention worked, for which individuals, and to what extent. Our findings have implications for decision-makers in fields like education, healthcare, law, and clinical practices concerning children and adolescents.