Association for Public Policy Analysis & Management

Multisite randomized trials provide unique opportunities for investigating not only prevalent causal mechanisms but also how the mechanisms may manifest differently across sites due to important between-site differences. Yet there is a void in the statistical literature and application research on multisite causal mediation analysis. Under the potential outcomes causal framework, we conceptualize a joint distribution of the site-specific direct and indirect effects of treatment assignment. We present an innovative weighting strategy for identifying and estimating the population average indirect effect and direct effect that decompose the total treatment effect. Moreover, our strategy identifies and estimates the between-site variance and covariance of these causal effects. We apply the strategy to a re-analysis of data from the National Job Corps Study, examining the program theory of Job Corps that intends to promote economic independence of disadvantaged youths primarily through education and vocational training. Using the annual earnings in the fourth year after randomization, we have found the following results:

(1)   Individual participation in education and training consistently mediated the program impact on earnings across all the experimental sites. In other words, the program impacts mediated by education and training (i.e., the indirect effect) did not show substantial variation across the sites, suggesting that the quality of education and training was nearly uniform across the Job Corps centers.

(2)   Job Corps programs generated impacts on earnings through pathways other than education and training (i.e., the direct effect). Such impacts may be attributed to a comprehensive set of other services offered by Job Corps centers that were typically unavailable under the control condition. Interestingly, the direct effect varied by a great amount from site to site, explaining the between-site variation of the ITT effect. This result highlights the crucial role of supplementary services focusing on risk reduction for disadvantaged youth in addition to providing education and training.

(3)   Individual participation in Job Corps education and training generated more benefits than education and training under the control condition. This evidence suggests that Job Corps programs produced a greater return to education and training than the alternative programs utilized by those in the control group.

(4)   Job Corps programs that successfully increased employment and earnings through education and training did not necessarily display success in doing so through other service provision.

We have developed a ratio-of-mediator-probability weighting (RMPW) method that identifies the causal parameters when the treatment assignment and the mediator value assignment under each treatment are strongly ignorable within each site given the observed pretreatment covariates. A method-of-moments procedure incorporating RMPW consistently estimates all these causal parameters. Unlike maximum likelihood based multilevel path analysis and structural equation modeling (SEM), our strategy conveniently relaxes the assumption of no treatment-by-mediator interaction while greatly simplifying the outcome model specification without invoking strong distributional assumptions. In addition, we derive asymptotic standard error estimators that reflect the sampling variability of the estimated weights. Simulation studies have shown satisfactory performance of these estimation procedures.