Association for Public Policy Analysis & Management

Low-income populations are so diverse that one-size-fits-all strategies to improve labor-market outcomes for members of these populations are unlikely to succeed. Identifying the context in which particular strategies work best can help practitioners and policymakers tailor policies to specific populations and goals. In this study, we used Bayesian meta-regression to synthesize the literature on employment and training interventions to learn not only what works, but also what works for whom and in what context.

Meta-regression applies traditional regression techniques to a data set comprising estimated effects; in this study, the estimated effects were the impacts of employment and training interventions examined in the Employment Strategies for Low-Income Adults Evidence Review (ESER), a systematic review focused on efforts to improve labor market outcomes for low-income workers. Treating these impact estimates as outcomes in a regression model allows us to summarize the sizes of the impacts estimated from different studies and to determine whether certain intervention features, like specific employment strategies, are associated with larger or smaller impacts.

We implemented this study using Bayesian methods, which combine data with structured assumptions to improve the precision and plausibility of the results. For example, our meta-regression model assumed that, in the absence of strong evidence to the contrary from the data, each employment strategy was equally effective. These assumptions enhanced the precision of the analysis by “borrowing strength” – drawing on information from precisely-estimated relationships to inform less-precisely-estimated relationships – across strategies. By improving the precision of our estimates of relationships among many factors, borrowing strength greatly increased the flexibility and resolution of our analysis. With this approach, we were able to investigate both which strategies were most effective overall and which strategies were most effective at improving specific types of outcomes. For example, although work experience is not statistically significantly associated with overall improvements in outcomes, this strategy is associated with statistically significant gains in short-term independence from public assistance.

Placing data in the context of these structured assumptions also allows us to draw probabilistic conclusions, which are not valid using conventional methods. Probabilistic language describes both the strength and magnitude of the results in natural, intuitive terms like “there is a 1 percent chance that financial incentives and sanctions improve impacts by 5 percent or more.” With this clear and flexible framing of the results, policy-makers can gain a deeper understanding of not just whether an intervention improves participants’ outcomes, but by how much.

In this study, we concluded that although almost all the employment strategies investigated in the systematic review improved participants’ outcomes, the improvements were modest. No single strategy led to dramatic gains in outcomes, suggesting that combinations of strategies are more likely to help low-income workers find and keep jobs than any strategy alone. The weak associations between strategies and outcomes only underscore the important role that other factors, such as implementation, may play in improving outcomes for these populations and reaffirms the importance of further qualitative and quantitative investigation.