Sol Stern’s new article on the Reading First study shenanigans offers a window into the central challenge of randomized experiments in education. That challenge is the violation of the Stable Unit Treatment Value Assumption (SUTVA) required for clean causal inference in randomized experiments. As articulated by ninja statistician Don Rubin, the most common violation of SUTVA involves “interference between units.”
What does interference mean? The idea is that Serena’s outcome should not be affected by whether her peers Blair and Vanessa were assigned to the treatment or control condition. In other words, one subject’s outcome should depend only on the treatment to which that subject is assigned, not on the treatment assignments of other subjects. But in many social settings, there are peer effects and social spillovers and, as a result, others’ treatment assignments likely do affect one’s own outcomes. Vaccinations provide another clear example - my risk of contracting a disease is dependent on my exposure, which is in turn affected by whether others have been vaccinated. In the case of Reading First, one school’s treatment likely affected what other schools in the district did, as Stern details in his article.
The lesson here is not just about the Reading First study. Experiments in social science are fundamentally different than experiments in medicine, and it turns out gold standard is often more silver or bronze than we would have hoped. Don’t get me wrong - I still dig experiments - but I’m not counting on them to solve all of the problems that vex our schools.