Unpacking Between-Site Heterogeneity in Complex Mediation Mechanisms in the Job Corps Impact
*Names in bold indicate Presenter
We develop a new analytic procedure aimed at enhancing both the internal validity and external validity of multisite causal mediation analysis. We consider vocational training and general education as two concurrent mediators central to the program, distinguish the relative contribution of each, and determine whether these two types of human capital investments reinforce each other’s mediating role. Under the potential outcomes causal framework, we decompose the total Job Corps impact into an indirect effect transmitted through vocational training, an indirect effect transmitted through general education, and a direct effect attributable to supplementary services. To identify not only the population average but also the between-site variance of the causal effects, we develop a ratio-of-mediator-probability weighting (RMPW) method. The identification is based on the assumptions that the treatment assignment and the mediator value assignment under each treatment are strongly ignorable, and the two mediators are independent across treatment conditions, within each site given the observed pretreatment covariates. To generalize the causal conclusions to the study population of eligible Job Corps applicants, we use a sample weight to adjust for sample and survey designs. We further estimate a nonresponse weight to account for non-random attrition through a propensity scoring approach.
To estimate the causal parameters, we develop a method-of-moments procedure. Unlike maximum likelihood based multilevel path analysis and structural equation modeling (SEM), our strategy greatly simplifies the outcome model specification without invoking strong distributional assumptions. The estimation of both the RMPW and non-response weight poses a challenge to statistical inference. We derive asymptotic standard error estimators that reflect the sampling variability of the estimated weights. Simulation studies have shown satisfactory performance of the estimation procedure. We have published an open-source R package, MultisiteMediation, offering applied researchers a convenient implementation tool.
The results reveal that for high school dropouts, the mediating role of vocational training becomes evident only when general education is improved. Moreover, the program impact varies across the Job Corps centers largely due to the variation in the direct effect transmitted through other supplementary services. In addition, based on a novel weighting-based sensitivity analysis strategy, we demonstrate that the analytic conclusion would not be easily altered by an unmeasured confounder. Such evidence is crucial for improving program implementation.