Randomized control trials (RCTs) are the “gold standard” evaluation method for assessing the effectiveness of a program or intervention (“treatment”). In experimental studies, researchers are interested in secondary questions that aim to explore the channels or mechanisms (“mediators”) through which programs operate. Such questions, however, cannot be addressed within the experimental framework because participants are generally not randomized to different levels of mediators, i.e., mediators are affected by the program; hence, they are endogenous.  

Four non-experimental approaches are typically used to study mediators in RCTs (Imai, Keele, Tingley, & Yamamoto; 2011; Jo, Stuart, MacKinnon, Vinokur, 2011; Jo, 2008; Sobel, 2008). The first approach, structural equation modeling (SEM), decomposes the treatment effect into a direct treatment effect and indirect effects realized through mediators where the latter consist of the treatment effects on mediators and mediator effects on the outcome.  Imai et al. (2011) refine the SEM approach and propose an alternative nonparametric identification strategy that allows the estimation of direct and indirect effects under less restrictive and more easily interpretable assumptions. The third approach, principal stratification, entails dividing treatment and control individuals into strata based on potential values of mediators (e.g., strata based on whether individuals’ mediator values are improved by the treatment or not). Separate treatment effects are estimated for each stratum, which are then used to assess mediator effects. The fourth approach, instrumental variables (IV) analysis, utilizes the exogenous portion of the variation in mediators created by the treatment to estimate mediator effects.  It is obvious that these approaches rely on different assumptions. For example, the first two approaches primarily depend on the ignorability of the observed mediator values given pre-treatment covariates while principal stratification invokes the ignorability of potential mediator values given pre-treatment covariates that predict stratum membership and IV relies on an exclusion restriction assumption, which implies that the treatment effect is completely realized through mediators (no direct treatment effect). 

The proposed paper presents the comparative assessment of these four approaches, focusing on their assumptions and estimands, and implements them to examine mediators in the experimental study of North Carolina's Early College High Schools (ECHS). ECHS is a new and rapidly spreading high school model designed to increase the number of students who graduate from high school and enroll and succeed in postsecondary education. The study uses an experimental design in which students assigned to the ECHS are selected from a pool of applicants via random assignment, while the control group consists of applicants who attend other high schools in the district. Analyses of ninth grade outcomes show that ECHS had a positive and statistically significant impact on taking and succeeding in core college preparatory course as well as attendance and suspension rates. This paper explores whether these effects can be explained through potential mediators including students’ self-efficacy in English and Math, engagement with school and coursework, and relationships with teachers; academic and social support provided to students; and teacher expectations. Analyses are underway and will be completed by the time of the conference.