Association for Public Policy Analysis & Management

Panel: Multi-Armed Trials for Informing Policy and Practice: What We’Ve Learned from 50 Years of Research

(Methods and Tools of Analysis)

The proposed panel will convene researchers who over the past several decades have pioneered multi-armed randomized experiments for social policy setting. Each participant, drawing on their own experiences in early efforts, and since then, will describe technical, pragmatic and political challenges to mounting multi-armed trials. The retrospectives will reflect on the policy contexts that led to research questions for which the multi-armed design alternatives were best-suited.

Participants will address the early concerns that led to cautious applications and expansion of the design. They will discuss opportunities multi-armed designs opened-up for addressing different but related policy-relevant questions such as concerning impacts of different treatments against each other, or net impact of each, or concerning the benefits of using additional components with a common base treatment, as opposed to comparing impacts of whole program alternatives. Panelists will also address the costs of the design, both material and design-related, including issues of statistical power associated with estimation of differences in impact between treatment arms, as opposed to the net impacts of each treatment, and with drawing multiple comparisons that multi-armed trials necessarily afford.

The panel will also focus on the pragmatics of moving from considerations of initial design to actual implementation, for instance, what it takes to get public agency staff to appreciate the benefits of specific multi-armed designs and go along with their use.

Presenters will also discuss how, in hindsight, specific designs may have been executed differently to address deficits that arose or to yield more information than was obtained. They will also consider the applicability of specific results to contemporary policy, given changes in the larger environment.

The multi-armed design alternative while simple in principle has led to complex new approaches. But as new variants, extensions and applications are introduced it is very important “to know where we’re coming from so that we know where we’re going.” The proposed panel will review the foundations and development of the design to enable those either using or planning the use of multi-arm trials – as conventionally designed and in the newest applications –  to get the most out of them, to understand the issues that may lead public agency officials and other stake-holder to become hesitant, and to continue to make the most use of past learnings and avoid pitfalls.