Association for Public Policy Analysis & Management

Social-psychological interventions bolster students’ social belonging by providing students with non-threatening ways to interpret the initial struggles during transition to college and have significantly improved disadvantaged students’ educational outcomes in several single-site field experiments (Walton and Cohen, 2011; Yeager et al., 2017; Murphy, Gopalan et al., In prep). To understand the robustness of these effects across different educational settings, we implemented a multi-site randomized control trial (RCT). This RCT included customization and replication of the social belonging interventions in 23 different educational institutions. This paper contributes to new understanding regarding the consequences of differential compliance for heterogeneity in treatment effects based on data from two sites.

Incoming first-year students received the online intervention as part of other pre-orientation tasks. Students were randomly assigned to one of three groups—two treatment (T1 and T2) and a control condition. Students assigned to the treatment conditions read stories from upperclassmen that encouraged them to attribute early academic and social struggles more productively, thereby bolstering students’ sense of social belonging. In T1, the upperclassmen stories were largely standardized across sites, whereas in T2 the stories were drawn from students’ experiences at the specific campus. Control group students read stories about adapting to the physical surroundings of the campus. In one of the sites, we implemented a pure control condition—students assigned to this group did not read anyupperclassmen stories.

Part of this paper’s main contribution is that it will estimate the differential effects of being assigned to a treatment versus receivingthe treatment. Defining “compliance” in a social-psychological intervention is not straightforward. Does “compliance” involve simply clicking on the online activity link, or engagement with the treatment message as measured by time spent on the activity or by students’ internalization of the treatment message as measured by students’ responses to manipulation checks? We define “compliers” using novel approaches—to our knowledge, the first in this literature—to further understand these contextual moderators of the treatment effects. Because “compliance” rates vary across sites, this analysis can elucidate important differences in the contexts within which the interventions are implemented (such as differential comprehension of treatment message, differential engagement with campus resources that bolster social belonging, and others) that may further moderate the efficacy of the treatment. Two-stage-least-squares (TSLS) estimation using instrumental variable approaches will then estimate the average treatment-on-the-treated (ATT or TOT) effect.

Preliminary analysis of the first cohort of students at one of the sites (in which students who clicked on the intervention link are included as compliers) show no evidence of statistically significant treatment effects on students’ academic or psychological outcomes. This paper will include persistence and academic data analysis from the second site as well. Finally, it is important to note that treatment effects may emerge over time with the inclusion of the second cohort of students, as that cohort will add substantial statistical power. These results highlight the need for new empirical techniques and evidence to elucidate the conditions necessary to realize the potential of these interventions at scale.