Panel: Confronting Challenges of Evaluation Designs That Consider “What Works Best for Whom?”
(Employment & Training)

Saturday, November 10, 2012: 8:30 AM-10:00 AM
Washington (Sheraton Baltimore City Center Hotel)

*Names in bold indicate Presenter

Organizers:  Laura Peck, Abt Associates, Inc.
Moderators:  Demetra Nightingale, U.S. Department of Labor and Jeffrey Smith, University of Michigan
Chairs:  Seth Chamberlain, Administration for Children and Families

Increasingly, federal agencies evaluating their social interventions – as well as state and local governments and foundation sponsors – are interested in more than the average treatment effect on the whole population of disadvantaged families or individuals served. The more interesting impact questions involve narrower populations (e.g., welfare recipients, high school dropouts, non-custodial parents) and pulling apart interventions to understand what specifically about them drives program impacts. Moreover, the average treatment effect may appear indistinguishable from zero, with the average masking potentially important effects from various parts of a multifaceted treatment or for specific subgroups. This panel includes three papers that think creatively about how we can answer “what works best for whom?” questions. The first explores new design options for unpacking the effects of various job search strategies; the second discusses designs for identifying causal mechanisms; and the third offers a new paradigm for social experimentation. Together we believe these insights, along with discussant commentary, will be of interest to the APPAM audience, given its interest in employment and training, social policy, experimental evaluation and causal inference therein.

New Evaluation Design Options for Assessing the Effectiveness of Job Search
Laura Peck, Abt Associates, Inc. and Stephen Bell, Westat

Causal Mediation Analysis, with Application to Job Search Intervention
Kosuke Imai, Princeton University, Luke Keele, Pennsylvania State University and Dustin Tingley, Harvard University

See more of: Employment & Training
See more of: Panel