Association for Public Policy Analysis & Management

Roundtable: Practical Tips and Advice for Launching an RCT

(Tools of Analysis: Methods, Data, Informatics and Research Design)

RCTs offer a credible, rigorous method of conducting program evaluation. There are numerous examples of how RCTs have shifted the national debate around certain subjects (early education, health insurance) and have led to the adoption of better evidence-based policy. Yet, many government agencies, non-profits and researchers in certain subfields have not yet used RCTs to answer critical policy questions. In some cases, unfamiliarity with the details of actually launching an RCT have stopped rigorous evaluations before they could get started. In fact, anyone who is thinking about doing an RCT will often have to overcome several specific, unexpected hurdles on the way to actually pulling one off. This talk will discuss the most common pitfalls, objections, and stumbling blocks that have the potential to derail the completion of RCTs. Roundtable participants who have worked on a variety of projects in the United States and in other countries will share their experience with RCTs and present the audience with both an overview of the major barriers to completing projects and well as practical, actionable advice for getting around those barriers. Roundtable participants include: • An academic researcher who has worked with U.S. government and non-profit partners to turn ideas for interventions into evaluations. • An implementing partner who has worked with researchers to embed an RCT into the design of their program. • A research manager who has launched several large-scale RCTs including in a major U.S. hospital and with a large social service non-profit. • A policy outreach manager who has advised major government agencies and school systems how they could set up research projects, including RCTs, within the constraints of their situations. • J-PAL’s Global Director of Research and Training, who will moderate the panel and provide international perspectives on how challenges can arise and be overcome in very difficult situations. Topics will include: • Innovative strategies to randomize participants to programs. • Working with entitlement programs where participants can’t be denied opportunities to participate. • Data limitations, privacy concerns, and managing administrative data challenges. • Institutional review board and data use agreement advice. • Advocating for RCTs in political circumstances. • How to design RCTs that won’t make governments or other implementing partners nervous. N.B. We have discussed this session with the White House Social and Behavioral Sciences Team, which will be sending staff to the conference but they are not yet sure which staff will be attending.