Association for Public Policy Analysis & Management

The literature on design-based methods has focused on RCTs with a single treatment and a single control group (Imbens and Rubin, 2015; Schochet, 2015, 2016). This theory, however, has not been formally extended to designs with multiple research groups. This is an important gap in the literature because multi-armed RCTs can simultaneously examine the effects of multiple interventions in a single study, thereby increasing the amount that researchers and policymakers can learn from impact studies. In social policy research, these designs are particularly relevant for interventions that are relatively easy to implement—for example, an education RCT testing several texting initiatives to improve student engagement and achievement. Relatedly, multi-armed designs are useful for rapid-cycle or opportunistic experiments aimed at continuous program improvement, for example, using behavioral-based interventions and encouragement designs.

This presentation will discuss results from a paper that addresses several key topics for estimating average treatment effects (ATEs) for multi-armed designs using design-based methods. In particular, the presentation will focus on the key question: How do design-based ATE estimators for the two-group design need to be modified for the multi-armed design when comparing pairs of research groups to each other?

As formalized in the paper, key components of the design-based theory for the two-group design apply also to the multi-armed context. However, two simple modifications are required:

1. Under the finite-population (FP) model where impact finings are to be generlatized to the study sample only, ATE estimators for each pairwise contrast pertain to the entire randomized sample, not just to the two groups being compared. Thus, variance estimators for the FP model for the two-group design need to be adjusted slightly to reflect the broader inference population.
2. For similar reasons, for blocked FP designs, weights for aggregating the block-specific impact estimates need to be modified to reflect the size of the full randomized sample in each block.

The presentation will also briefly address several other topics related to multi-armed trials:

• What multiple comparison adjustments should be used when conducting hypothesis tests across pairwise contrasts to identify the most effective interventions? The presentation will provide a brief summary of adjustment methods that align with the non-parametric underpinnings of the design-based approach with some simulation evidence.

• What assumptions are required to identify and estimate the complier average causal effect (CACE) parameter for multi-armed RCTs? The presentation will briefly show using an instrumental variable (IV) framework, that identification becomes much more complex in the multi-armed context, and may not be possible in some cases.