Association for Public Policy Analysis & Management

Roundtable: Implementation Research Using Multiple Methods in Large-Scale, Multi-Site Randomized Controlled Trials

(Methods and Tools of Analysis)

This roundtable focuses on implementation research (IR) conducted in large-scale, multi-site randomized controlled trial studies. For example:

• IR in the Mother and Infant Home Visiting Program Evaluation (MIHOPE) analyzed information from home visitors and local staff at 88 local programs, and for over 2,000 families that were randomly assigned to the MIHOPE program group. It collected information from family surveys and observations of families’ home environments at baseline, family service logs, observations of home visitor-family interactions, staff surveys, semi-structured qualitative interviews with home visitors, training logs, supervision logs, inventories of community services, surveys and interviews with evidence-based model developers, and reviews of local program and evidence-based model documents.
• IR in the National Implementation Evaluation of the Health Profession Opportunity Grants (HPOG) (49 distinct programs and over 30,000 participants) and the HPOG 1.0 Impact Study (42 programs and a sample of about 14,000 treatment and control group members) analyzed information from informant surveys, sites visits, follow-up surveys, program administrative data, and various secondary sources .

IR in large-scale, multi-site RCTs can provide rich opportunities for exploring questions directly relevant to this year’s conference theme of “Evidence for Action.” For example, these studies inform how local programs implement evidence-based programs, how implementation varies across and within programs and what characteristics are associated with varied implementation; how systems change occurs; frontline providers’ perceptions of their experiences; and the challenges that local programs face in implementing evidence-based programs.

This roundtable will address these topics as well as questions such as:

1. What kinds of research questions can best be answered by IR in large-scale, multi-site RCTs?
2. What are unique opportunities and challenges of conducting IR in many sites?
3. What are additional opportunities and challenges of conducting IR in an RCT study conducted across a number of sites (for example, analyzing implementation of programs available to or used by control group members)?
4. What are “lessons learned” about research design and analysis from IR studies in large, multi-site RCTs?
5. What kinds of promising approaches to IR should be considered for future large-scale, multi-site RCTs?
6. What IR designs are “low-return” and might be de-emphasized in future studies?

Panelists for the roundtable include two former federal agency leaders in program evaluation; and two project directors of large-scale, multi-site studies that feature multiple-method implementation research.