Association for Public Policy Analysis & Management

Examining the effectiveness of policies with spillovers to non-participants proves difficult. Since treatment may also influence control outcomes, program spillover can invalidate the treatment-control comparison even in randomized control trials (RCTs).  Yet, the effectiveness of a program may also rely on spillover, or peer effects, as a pathway or mechanism for inducing positive change.  In such cases, research designs that allow for the estimation of direct and spillover effects are necessary for determining program effectiveness.  In this study, we implemented an RCT at student, grade, and school levels in order to measure both direct and indirect, or spillover effects, of a policy.

Specifically, we examine the effectiveness of Check & Connect (C&C) a student-centered mentoring intervention for high-absence students aimed at improving school engagement and attendance.  This paper measures the direct and spillover effects of C&C on eighth-grade students’ high school choice participation - a proxy for school engagement.  Poor application strategies, limited adult support, and logistical concerns appear to create enrollment barriers to choice schools in Chicago (Stevens, de la Torre, & Johnson, 2011).  Mentors may directly mitigate these barriers by providing students with information and encouragement as well as individualized support, which counselors rarely offer.  Mentor-parent connections also may lead to increased parental involvement in the choice process.

Moreover, randomly selected control students who attend schools with C&C mentors may be indirectly influenced by C&C via the school choice actions of their peers (i.e. peer effects) or other changes mentors have on the school environment.  Social spillover might be particular important in the school choice context, since adolescents appear particularly prone to peer influence and must use limited academic and social information to make this complex one-time schooling decision.  Thus, a simple treatment-control RCT comparison in this context might yield misleading measures of program effectiveness given the likelihood of positive spillovers to control students at C&C treatment schools.

In fact, quantitative analysis finds only small non-significant differences in school choice participation between treatment and control students who attend the same school. Yet, substantial spillover appears to mask program effectiveness.  In contrast, a school-level comparison between students at C&C treatment and control schools finds that attending a C&C school induces students to submit 1.5 more applications (p < 0.01) and increases likelihood of acceptance by 8.6 percentage points (p < 0.01).  Moreover, these school-level impacts are reduced only slightly when just considering spillover effects.  Control student at C&C treatment schools apply to 1.4 more schools (p < 0.01) and are 7.6 percentage points more likely to be accepted (p < 0.05) than students who attend C&C control schools.  Preliminary qualitative analysis of semi-structured interviews with mentors and students, however, suggests variation in the extent that mentors help students navigate the choice process and interact with control students.

Stevens, W. D., de la Torre, M., & Johnson, D. (2011). Barriers to Access: High School Choice Process and Outcomes in Chicago. In School choice and school improvement (pp. 125–146). Cambridge, MA: Harvard Education Press.