Association for Public Policy Analysis & Management

Job Corps is the nation’s largest education and training program for 16-24 year old disadvantaged youths, most of whom have dropped out of high school. It intends to promote participants’ economic independence primarily through education and vocational training along with a wide array of services for risk reduction. The National Job Corps Study evaluated the program through a multisite randomized trial in which individuals were assigned to different treatment conditions within each site. The multisite design provides a unique opportunity for investigating between-site heterogeneity in the mediation mechanism that characterizes the educational process central to the program theory. However, such opportunities have not been fully utilized due to some important constraints of existing analytic tools.

We develop a new analytic procedure aimed at enhancing both the internal validity and external validity of multisite causal mediation analysis. We consider vocational training and general education as two concurrent mediators central to the program, distinguish the relative contribution of each, and determine whether these two types of human capital investments reinforce each other’s mediating role. Under the potential outcomes causal framework, we decompose the total Job Corps impact into an indirect effect transmitted through vocational training, an indirect effect transmitted through general education, and a direct effect attributable to supplementary services. To identify not only the population average but also the between-site variance of the causal effects, we develop a ratio-of-mediator-probability weighting (RMPW) method. The identification is based on the assumptions that the treatment assignment and the mediator value assignment under each treatment are strongly ignorable, and the two mediators are independent across treatment conditions, within each site given the observed pretreatment covariates. To generalize the causal conclusions to the study population of eligible Job Corps applicants, we use a sample weight to adjust for sample and survey designs. We further estimate a nonresponse weight to account for non-random attrition through a propensity scoring approach.

To estimate the causal parameters, we develop a method-of-moments procedure. Unlike maximum likelihood based multilevel path analysis and structural equation modeling (SEM), our strategy greatly simplifies the outcome model specification without invoking strong distributional assumptions. The estimation of both the RMPW and non-response weight poses a challenge to statistical inference. We derive asymptotic standard error estimators that reflect the sampling variability of the estimated weights. Simulation studies have shown satisfactory performance of the estimation procedure. We have published an open-source R package, MultisiteMediation, offering applied researchers a convenient implementation tool.

The results reveal that for high school dropouts, the mediating role of vocational training becomes evident only when general education is improved. Moreover, the program impact varies across the Job Corps centers largely due to the variation in the direct effect transmitted through other supplementary services. In addition, based on a novel weighting-based sensitivity analysis strategy, we demonstrate that the analytic conclusion would not be easily altered by an unmeasured confounder. Such evidence is crucial for improving program implementation.