Association for Public Policy Analysis & Management

This paper will draw on Mathematica’s experiences designing and implementing demonstration studies for the U.S. Department of Labor to discuss how “on-the-ground” realities shape study design, implementation, and results. When designing pilot programs, study teams are faced with the need to balance innovation with practicality, analytical rigor with feasibility. Four important issues frequently arise within such efforts: 1) something new is being tried to see if it will work; this requires persuading local partners to depart from business as usual and to implement the specified model with fidelity; 2) it is often sought to implement a uniform model across sites with varied and evolving local contexts about which, at the outset, imperfect knowledge is available; 3) implementation success frequently hinges on local partners continuing to prioritize a program that is temporary while they are still responsible, and directly accountable, for implementing numerous other regular duties; and 4) to effectively evaluate these efforts, frequently local partners must also agree to modify their intake procedures and abide by random assignment results, as well as cooperate with other aspects of the evaluation (e.g., implementation and outcomes data collection). 

We will draw on lessons from the Individual Training Account Experiment, Self-Employment Training Demonstration, and other random assignment demonstrations to discuss the kinds of decisions and important that policymakers and researchers commonly negotiate, and how these can shape the design, implementation, and results of demonstrations studies.