The large number of multi-site randomized trials and regression discontinuity analyses that were conducted during the past decade have produced opportunities for researchers to study the causal factors that produce or “mediate” the intervention effects that have been observed.  One widely used approach for this type of analysis is instrumental variables (IV) analysis. With a multi-site randomized trial or a regression discontinuity design, it is possible to create multiple instrument variables (Site-specific instruments can be constructed by interacting treatment assignment with a zero/one indicator for each site). These instrumental variables can be used to study mediator effect variation and, potentially, effects of multiple mediators.  This approach can have both advantages and disadvantages.

The present paper explores these potential advantages and disadvantages. In particular, it explores the use of instrumental variables (IV) analysis with a multi-site randomized trial or regression discontinuity design to estimate the effect of a mediating variable on an outcome. To do so, the paper uses a random-coefficient IV model that allows the impact of program assignment on the mediator (compliance with assignment) and the impact of the mediator on the outcome (the treatment effect) to vary randomly across sites and thus co-vary. This realistic extension of conventional fixed-coefficient IV analysis brings to light a new potential bias in IV analysis which Reardon and Raudenbush (2011) refer to as “compliance-effect covariance bias.”

 The first part of the paper examines algebraically and through simulation the implications of this bias for point estimates and estimated standard errors from conventional fixed coefficient two-stage least squares (2SLS) estimates of mediator effects and compares these findings with those from a traditional OLS analysis of mediator effects. The next part of the paper develops an estimator that corrects for compliance-effect covariance bias and compares its properties to those of 2SLS and OLS. When the first stage F-statistic of 2SLS exceeds 10 (a commonly-used threshold for instrument strength) the bias-corrected estimator typically performs better than 2SLS or OLS. The final part of the paper uses the new estimator and 2SLS to study mediator effects from two empirical examples: (1) the effect of class-size on student academic achievement in math and reading using data from the Tennessee class size experiment, Project STAR, and (2) the effect of scientifically-based reading instruction on student reading achievement using data from the federal Reading First Impact Study.