Association for Public Policy Analysis & Management

This paper investigates the extent to which disadvantaged workers who get a larger, potentially better “dose” of a transitional employment intervention experience relatively larger impacts. It extends the Analysis of Symmetrically-Predicted Endogenous Subgroups (ASPES) method to derive lower and upper bounds on impacts to get robust policy findings. Hypothesis tests for the bounded estimates examine whether each particular subgroup of intervention participants—from four subgroups defined by the speed of entry into and duration in subsidized jobs—benefited, and whether one subgroup benefited more than another. Across six outcome measures, 60 lower bound estimates of impact were produced and 60 upper bound estimates. While all the estimates contained bias, remarkably the analysis produced results highly conclusive for policy. For 18 of the 60 outcomes (30 percent) unambiguous and statistically significant evidence emerged that the transitional jobs intervention improved outcomes for a subgroup or did so for one subgroup more than another (these are the instances with both lower and upper bound estimates on the same side of 0 and the estimate nearer 0 statistically significant). The research demonstrates that—when properly tested and interpreted for policy—bounding strategies in impact analysis can be as effective as conventional approaches. Evaluators are encouraged to develop and apply bounded approaches when randomized control trials free from downward and upward bias cannot be implemented.